Based on this discussion and our team’s evaluation, I want to share our decision framework and key insights that might help others facing similar choices.
Our Decision: Hybrid Approach
We’re implementing HashiCorp Vault as our primary secrets management platform with Azure Key Vault for Azure-specific managed identities and certificates. This hybrid approach addresses all three critical requirements: secrets rotation, audit logging, and multi-cloud portability.
Secrets Rotation and Audit Logs:
Vault’s dynamic secrets capability is transformative for database credentials in our procurement module. Instead of long-lived credentials, Vault generates time-limited credentials on-demand with automatic revocation. Our implementation:
- Database credentials: 24-hour TTL with automatic rotation
- API keys for supplier integration: 7-day TTL with manual rotation capability
- OAuth tokens: Vault acts as OAuth client, refreshing tokens automatically
- Service principal secrets: 90-day rotation with 15-day grace period
For audit logging, we configured Vault to stream audit logs to our centralized SIEM (Splunk). Every secret access, rotation, and policy change is logged with full context including GitLab job ID, user identity, and timestamp. This satisfies our SOC2 compliance requirements. Azure Key Vault logs flow to Azure Monitor for Azure-specific resources.
GitLab CI/CD Integration:
We implemented JWT-based authentication between GitLab and Vault, which eliminates the need to store Vault tokens in GitLab. Each pipeline job authenticates using its JWT token, and Vault validates the token against our policy. Our GitLab pipeline configuration:
- CI/CD variables contain only Vault address and namespace
- Pipeline jobs authenticate using GitLab’s CI_JOB_JWT token
- Vault policies grant access based on GitLab project, branch, and environment
- Secrets are injected as environment variables at runtime, never written to disk
This approach provides fine-grained access control - our dev environment pipelines can’t access production secrets, and feature branches have read-only access to non-sensitive test credentials.
Multi-Cloud Portability:
Vault’s cloud-agnostic design was crucial for our multi-cloud strategy. We deployed Vault on Kubernetes using the official Helm chart, with Raft integrated storage for HA. The same Vault cluster serves both Azure and AWS deployments:
- Azure: Vault integrates with Azure AD for authentication, uses Azure Storage for backups
- AWS: Vault uses AWS IAM authentication, leverages AWS S3 for encrypted backup storage
- Consistent API and workflow across both clouds
- Secrets are namespaced by cloud provider and environment
For procurement module deployments, we use cloud-specific secret engines (Azure secrets engine for Azure resources, AWS secrets engine for AWS resources) which generate dynamic, short-lived cloud credentials. This eliminates static service principal credentials entirely.
Key Lessons Learned:
-
Start with policy design: We spent significant time designing Vault policies before implementation. Clear policy boundaries prevent security gaps and simplify troubleshooting.
-
Automate unseal process: Use Vault’s auto-unseal feature with cloud KMS (Azure Key Vault or AWS KMS). Manual unsealing is operationally challenging for HA deployments.
-
Implement secret versioning: Both Vault and Azure Key Vault support versioning. Our procurement services always reference latest version but can roll back if needed.
-
Monitor secret usage patterns: Unusual access patterns often indicate security issues or misconfigured applications. We alert on secrets accessed from unexpected IP ranges or outside business hours.
-
Plan for secret migration: Moving from GitLab variables to Vault required careful planning. We migrated environment by environment with parallel runs to validate behavior.
-
Document emergency procedures: When Vault is unavailable, deployments stop. We maintain encrypted emergency break-glass credentials for critical recovery scenarios.
Comparison Summary:
Choose HashiCorp Vault if:
- Multi-cloud deployments are required
- Dynamic secrets generation is valuable
- You need advanced secret workflows (approval processes, secret wrapping)
- Team has Kubernetes expertise for Vault hosting
Choose Azure Key Vault if:
- Azure-only infrastructure
- Deep integration with Azure services is priority
- Prefer fully managed service over self-hosted
- Team is Azure-focused with limited Kubernetes experience
Consider hybrid approach if:
- Multi-cloud but heavily Azure-invested
- Want managed identity benefits in Azure
- Need Vault’s dynamic secrets for databases
- Can manage operational complexity
Our procurement module deployments now have zero static secrets in source control or CI/CD variables, complete audit trails for compliance, and seamless multi-cloud operation. The initial investment in Vault setup has paid off in improved security posture and operational flexibility.