Secrets Management

​

Overview

​

Architecture

Vault (EC2) → vault-secret-sync (CronJob) → Kubernetes Secrets → Application Pods

​

Vault Server

• Instance Type: t3.small
• AMI: Official Vault OSS AMI
• Storage Backend: PostgreSQL (RDS)
• Auto-unseal: AWS KMS key
• Access: Application Load Balancer with TLS
• URL: https://vault.pennlabs.org

• Vault data is stored in a PostgreSQL database on RDS
• AWS KMS automatically unseals Vault on instance restart
• TLS certificate managed by AWS Certificate Manager
• Only accessible through load balancer (port 8200)

​

Secret Storage Location

secrets/production/default/<app-name>

secrets/production/default/db-backup

​

How Applications Access Secrets

​

1. Secrets Defined in Vault

resource "vault_generic_secret" "db-backup" {
  path = "${module.vault.secrets-path}/production/default/db-backup"

  data_json = jsonencode({
    "DATABASE_URL" = "postgres://..."
    "S3_BUCKET"    = "sql.pennlabs.org"
  })
}

​

2. vault-secret-sync Cronjob

namespaces:
  - default
  - monitoring

role_arn: ${role_arn}

1. Authenticates to Vault using AWS IAM role
2. Reads all secrets from configured Vault paths
3. Creates/updates Kubernetes Secret objects in specified namespaces
4. Runs on a schedule to keep secrets in sync

• Assume the secret-sync role in Vault
• Read secrets from the secrets path

​

3. Applications Reference Secrets

new DjangoApplication(this, 'django-asgi', {
  deployment: {
    image: backendImage,
    secret: 'my-app', // References the Kubernetes secret
    env: [
      { name: 'REDIS_HOST', value: 'redis' },
    ],
  },
});

​

Database Secrets

module "db-secret-flush" {
  source = "./modules/vault_flush"
  for_each = setsubtract(local.database_users, ["vault"])
  path     = "secrets/production/default/${each.key}"
  entry    = { 
    DATABASE_URL = "postgres://${each.key}:${postgresql_role.role[each.key].password}@${aws_db_instance.production.endpoint}/${each.key}" 
  }
}

• Username
• Password (generated by Terraform)
• Database endpoint
• Database name

​

Vault Access Control

​

Who Has Access?

​

IAM Authentication

resource "vault_policy" "secret-sync" {
  name = "secret-sync"
  policy = templatefile("${path.module}/policies/secret-sync.hcl", {
    PATH = vault_mount.secrets.path
  })
}

resource "vault_aws_auth_backend_role" "secret-sync" {
  role                     = "secret-sync"
  bound_iam_principal_arns = [var.SECRET_SYNC_ARN]
  token_policies           = [vault_policy.secret-sync.name]
}

​

Managing Secrets

​

Adding a New Secret

1. Add secret definition to terraform/vault.tf:
   resource "vault_generic_secret" "my-app" {
     path = "${module.vault.secrets-path}/production/default/my-app"
     
     data_json = jsonencode({
       "API_KEY" = var.MY_APP_API_KEY
       "SECRET_KEY" = var.MY_APP_SECRET_KEY
     })
   }
   
2. Add variables to terraform/variables.tf:
   variable "MY_APP_API_KEY" {
     type = string
     sensitive = true
   }
   
3. Set the variable in your local .env file
4. Apply the change:
   source .env
   terraform plan  # Preview changes
   terraform apply -target=vault_generic_secret.my-app
   

1. Log into Vault at https://vault.pennlabs.org
2. Navigate to the secrets engine
3. Create/edit secrets at secrets/production/default/<app-name>
4. Wait for next sync cycle or manually trigger sync

​

Updating an Existing Secret

1. Update in Vault (via Terraform or UI)
2. Wait for sync - vault-secret-sync will update the Kubernetes Secret
3. Restart pods to pick up new values:
   kubectl rollout restart deployment/<deployment-name>
   

​

Deleting a Secret

1. Remove from Vault (Terraform or UI)
2. Delete the Kubernetes Secret:
   kubectl delete secret <secret-name>
   
3. Update application deployment to not reference the secret

​

Team Sync

• Vault access - Team leads get Vault UI access
• Bitwarden access - Access to shared password vault
• Django admin - Admin console access for applicable apps

cronjobs:
  - name: team-sync
    schedule: "*/30 * * * *"  # Every 30 minutes
    secret: team-sync
    image: pennlabs/team-sync

module "team-sync-flush" {
  source = "./modules/vault_flush"
  path   = "${module.vault.secrets-path}/production/default/team-sync"
  
  entry = {
    GITHUB_TOKEN = var.GH_PERSONAL_TOKEN
  }
}

​

Security Best Practices

​

DO:

• ✓ Use Terraform to manage secrets when possible (version controlled, auditable)
• ✓ Use sensitive variables in Terraform for secret values
• ✓ Rotate secrets regularly, especially API keys
• ✓ Use scoped IAM roles (principle of least privilege)
• ✓ Monitor Vault audit logs
• ✓ Use strong, unique passwords for each service

​

DON’T:

• ✗ Commit secrets to Git repositories
• ✗ Share Vault tokens or root credentials
• ✗ Store secrets in ConfigMaps (use Secrets instead)
• ✗ Hard-code secrets in application code
• ✗ Use the same secret across multiple applications
• ✗ Give write access to automated systems unless necessary

​

Troubleshooting

​

Secret Not Appearing in Kubernetes

1. Check if secret exists in Vault:
   vault kv get secrets/production/default/<app-name>
   
2. Check vault-secret-sync logs:
   kubectl logs -l app=vault-secret-sync -n monitoring
   
3. Verify IAM role has permissions:
   • Check that the role ARN matches in Vault policy
   • Ensure role can be assumed by the service account
4. Manually trigger sync (if needed):
   kubectl create job --from=cronjob/vault-secret-sync manual-sync-1
   

​

Application Can’t Access Secret

1. Verify secret exists:
   kubectl get secret <secret-name>
   kubectl describe secret <secret-name>
   
2. Check pod has secretRef:
   kubectl get pod <pod-name> -o yaml | grep -A5 secretRef
   
3. Check for typos in secret name in deployment config
4. Restart pods after secret creation/update:
   kubectl rollout restart deployment/<deployment-name>
   

​

Vault Unsealing Issues

1. Check KMS key permissions
2. Verify Vault instance IAM role has KMS decrypt permission
3. Manually unseal if needed (requires unseal keys from initialization)

​

Additional Resources

• Vault Documentation
• Secrets Flow Documentation
• Database Backups