Skip to content

Advanced Topics

Learn about some security considerations, performance implications, migration strategies, troubleshooting tips, and best practices for secrets management in Open Horizon.

Security considerations

To manage secrets securely, you must consider the storage, access control, and transmission of the secrets.

Secret storage

  • Secrets are stored in OpenBao or HashiCorp Vault using the KV V2 secrets engine.
  • All secrets are encrypted at rest.
  • Access is controlled through OpenBao or Vault ACLs.
  • Secrets are isolated organization.

Access control

  • Organization admins have full control over organization-wide secrets.
  • Users can only access their own private secrets.
  • The agbot component has read-only access to secrets.
  • Exchange root users have no direct access to secrets.

Secret transmission

  • Secrets are transmitted securely between components.
  • Secrets are only sent to nodes that need them.
  • Secret updates are sent through secure channels.
  • All secret operations are logged.

Performance implications

Several secrets-related activities may impact application performance. Here are the three main areas to consider.

Secret updates

  • Secret updates trigger agreement updates.
  • You can batch multiple secret updates.
  • Follow best practices when updating services to ensure potential error conditions are addressed.
  • Consider update frequency and impact.

Storage considerations

  • OpenBao or Vault performance can be affected by:
  • Number of secrets
  • Secret size
  • Update frequency
  • Access patterns

Network impact

  • Secret updates require network communication.
  • Consider network bandwidth and latency.
  • Plan for offline scenarios.
  • Implement proper retry mechanisms.

Migration strategies

As an application is migrated to a secrets management solution, consider the following steps during the initial move and during application version upgrades.

Moving to secrets management

  1. Assessment
  2. Identify current secret storage
  3. Map secret usage
  4. Plan migration order

  5. Preparation

  6. Set up secrets manager
  7. Configure access controls
  8. Create backup strategy

  9. Migration

  10. Create new secrets
  11. Update service definitions
  12. Test in staging
  13. Roll out gradually

  14. Verification

  15. Verify secret access
  16. Test updates
  17. Monitor performance
  18. Clean up old storage

Version upgrades

  • Plan for backward compatibility
  • Test with new versions
  • Have rollback plan
  • Update documentation

Troubleshooting

When tracking down the source of problems, here are some areas to look and some tools to use.

Common issues

  1. Secret access problems
  2. Check OpenBao or Vault logs
  3. Verify ACLs
  4. Check network connectivity
  5. Verify authentication

  6. Update failures

  7. Check agreement status
  8. Verify service configuration
  9. Check network connectivity
  10. Review service logs

  11. Performance issues

  12. Monitor OpenBao or Vault metrics
  13. Check network latency
  14. Review access patterns
  15. Consider caching

Debugging tools

# Check secret status
hzn secretsmanager secret list

# View Vault logs
journalctl -u vault

# Check service logs
hzn service log -f <service>

# Monitor network
tcpdump -i any port 8200

Best practices

The following are some practices to consider as you use secrets management categorized by role.

Security

  1. Secret management
  2. Rotate secrets regularly
  3. Use strong encryption
  4. Implement proper access controls
  5. Monitor access patterns

  6. Service design

  7. Minimize secret usage
  8. Implement proper error handling
  9. Use secure communication
  10. Follow principle of least privilege

Operations

  1. Monitoring
  2. Track secret usage
  3. Monitor update frequency
  4. Watch for failed updates
  5. Track access patterns

  6. Maintenance

  7. Regular backups
  8. Performance optimization
  9. Clean up unused secrets
  10. Update operations documentation for each code change

Development

  1. Testing
  2. Test all secret operations
  3. Verify update handling
  4. Check error scenarios
  5. Test performance

  6. Documentation

  7. Document purpose and valid values
  8. Update procedures
  9. Troubleshooting guides
  10. Security considerations