Secrets Management

Overview

The GovTech platform uses AWS Secrets Manager for centralized, encrypted credential storage with automatic rotation and audit trails.

Never Hardcode Secrets:❌ WRONG:

const dbPassword = 'my-secret-password';  // NEVER DO THIS
const apiKey = process.env.API_KEY;  // Environment variables can leak

✅ CORRECT:

const AWS = require('aws-sdk');
const secretsManager = new AWS.SecretsManager();

const secret = await secretsManager
  .getSecretValue({ SecretId: 'govtech/prod/db-credentials' })
  .promise();

const dbCreds = JSON.parse(secret.SecretString);

Secret Storage Architecture

┌─────────────────────────────────────────────────────────────┐
│                    AWS Secrets Manager                       │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  govtech/prod/db-credentials  (KMS Encrypted)         │  │
│  │  govtech/prod/jwt-secret      (KMS Encrypted)         │  │
│  │  govtech/staging/db-credentials                       │  │
│  │  govtech/dev/db-credentials                           │  │
│  │  govtech/cicd/github-token                            │  │
│  └───────────────────────────────────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
                         │ IRSA (IAM Role for Service Account)
                         │ No hardcoded credentials needed
                         ▼
              ┌──────────────────────┐
              │  Backend Pod         │
              │  ServiceAccount:     │
              │  govtech-backend     │
              │                      │
              │  Annotation:         │
              │  eks.amazonaws.com/  │
              │  role-arn: arn:aws:  │
              │  iam::835960996869:  │
              │  role/govtech-       │
              │  backend-role        │
              └──────────────────────┘

AWS Secrets Manager

Secret Naming Convention

Secrets follow a hierarchical path structure:

{project}/{environment}/{secret-type}

Examples:

govtech/prod/db-credentials
govtech/prod/jwt-secret
govtech/staging/db-credentials
govtech/dev/db-credentials
govtech/cicd/github-token

This naming convention enables environment-based IAM policies that restrict access (e.g., developers can read govtech/dev/* but not govtech/prod/*).

Creating Secrets

AWS CLI
Terraform

# Create database credentials secret
aws secretsmanager create-secret \
  --name govtech/prod/db-credentials \
  --description "PostgreSQL credentials for production" \
  --kms-key-id alias/govtech-prod \
  --secret-string '{
    "username": "govtech_admin",
    "password": "SECURE_RANDOM_PASSWORD",
    "host": "govtech-prod-db.us-east-1.rds.amazonaws.com",
    "port": "5432",
    "dbname": "govtech"
  }'

# Create JWT secret
aws secretsmanager create-secret \
  --name govtech/prod/jwt-secret \
  --description "JWT signing key" \
  --kms-key-id alias/govtech-prod \
  --secret-string "$(openssl rand -base64 32)"

resource "aws_secretsmanager_secret" "db_credentials" {
  name                    = "govtech/prod/db-credentials"
  description             = "PostgreSQL credentials for production"
  kms_key_id              = aws_kms_key.govtech_main.arn
  recovery_window_in_days = 7

  tags = {
    Environment = "prod"
    Project     = "govtech"
    ManagedBy   = "terraform"
  }
}

resource "aws_secretsmanager_secret_version" "db_credentials" {
  secret_id = aws_secretsmanager_secret.db_credentials.id

  secret_string = jsonencode({
    username = "govtech_admin"
    password = random_password.db_password.result
    host     = aws_db_instance.postgres.endpoint
    port     = "5432"
    dbname   = "govtech"
  })
}

resource "random_password" "db_password" {
  length  = 32
  special = true
}

Updating Secrets

# Update secret value
aws secretsmanager put-secret-value \
  --secret-id govtech/prod/db-credentials \
  --secret-string '{
    "username": "govtech_admin",
    "password": "NEW_PASSWORD",
    "host": "govtech-prod-db.us-east-1.rds.amazonaws.com",
    "port": "5432",
    "dbname": "govtech"
  }'

# Verify update
aws secretsmanager describe-secret \
  --secret-id govtech/prod/db-credentials

Rolling Deployment Required:After updating a secret, restart pods to fetch the new value:

kubectl rollout restart deployment backend -n govtech

Or implement automatic secret rotation using AWS Lambda.

Retrieving Secrets

# Get secret value
aws secretsmanager get-secret-value \
  --secret-id govtech/prod/db-credentials \
  --query SecretString \
  --output text | jq .

# Output:
{
  "username": "govtech_admin",
  "password": "SECURE_PASSWORD",
  "host": "govtech-prod-db.us-east-1.rds.amazonaws.com",
  "port": "5432",
  "dbname": "govtech"
}

IRSA (IAM Roles for Service Accounts)

IRSA allows Kubernetes pods to assume IAM roles without hardcoded AWS credentials.

How IRSA Works

ServiceAccount with IAM Role Annotation

Create a Kubernetes ServiceAccount with an IAM role ARN:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: govtech-backend
  namespace: govtech
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::835960996869:role/govtech-backend-role
automountServiceAccountToken: true

IAM Role with Trust Policy

Create an IAM role that trusts the EKS cluster’s OIDC provider:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::835960996869:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:govtech:govtech-backend"
        }
      }
    }
  ]
}

Attach Secrets Manager Policy

Attach a policy allowing access to secrets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:835960996869:secret:govtech/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:835960996869:key/12345678-1234-1234-1234-123456789012"
    }
  ]
}

Deploy Pod with ServiceAccount

Reference the ServiceAccount in the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: govtech
spec:
  template:
    spec:
      serviceAccountName: govtech-backend  # Use IRSA-enabled SA
      containers:
        - name: backend
          image: 835960996869.dkr.ecr.us-east-1.amazonaws.com/govtech-backend:latest
          env:
            - name: AWS_REGION
              value: us-east-1

Backend Code with IRSA

const AWS = require('aws-sdk');

// No credentials needed - IRSA automatically provides them
const secretsManager = new AWS.SecretsManager({
  region: process.env.AWS_REGION || 'us-east-1'
});

async function getDbCredentials() {
  try {
    const data = await secretsManager
      .getSecretValue({ SecretId: 'govtech/prod/db-credentials' })
      .promise();
    
    return JSON.parse(data.SecretString);
  } catch (error) {
    console.error('Error retrieving secret:', error);
    throw error;
  }
}

// Usage
const dbCreds = await getDbCredentials();
const connectionString = `postgresql://${dbCreds.username}:${dbCreds.password}@${dbCreds.host}:${dbCreds.port}/${dbCreds.dbname}`;

Kubernetes Secrets (Fallback)

For non-production environments or secrets that don’t require AWS Secrets Manager:

Creating Kubernetes Secrets

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: govtech-secrets
  namespace: govtech
  labels:
    app: govtech
    component: secrets
type: Opaque
stringData:
  DATABASE_URL: "postgresql://user:[email protected]:5432/govtech_db"
  JWT_SECRET: "your-256-bit-secret"
  SESSION_SECRET: "your-session-secret"

Security Considerations:

Kubernetes secrets are base64 encoded, NOT encrypted by default

Enable encryption at rest for etcd:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>

Use AWS Secrets Manager for production secrets
Never commit secrets.yaml to Git (use secrets.yaml.template)

Using Secrets in Pods

Environment Variables
Mounted Files

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: backend
          image: backend:latest
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: govtech-secrets
                  key: DATABASE_URL
            - name: JWT_SECRET
              valueFrom:
                secretKeyRef:
                  name: govtech-secrets
                  key: JWT_SECRET

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: backend
          image: backend:latest
          volumeMounts:
            - name: secrets
              mountPath: "/etc/secrets"
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: govtech-secrets

Secrets available as files:

/etc/secrets/DATABASE_URL
/etc/secrets/JWT_SECRET

Secret Rotation

Automatic Rotation with Lambda

AWS Secrets Manager supports automatic rotation using Lambda functions:

# Enable rotation every 30 days
aws secretsmanager rotate-secret \
  --secret-id govtech/prod/db-credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:835960996869:function:SecretsManagerRDSPostgreSQLRotationSingleUser \
  --rotation-rules AutomaticallyAfterDays=30

Manual Rotation Process

Create New Password

NEW_PASSWORD=$(openssl rand -base64 32)
echo $NEW_PASSWORD  # Save this temporarily

Update Database

ALTER USER govtech_admin WITH PASSWORD 'NEW_PASSWORD';

Update Secret in AWS Secrets Manager

aws secretsmanager put-secret-value \
  --secret-id govtech/prod/db-credentials \
  --secret-string '{
    "username": "govtech_admin",
    "password": "NEW_PASSWORD",
    "host": "govtech-prod-db.us-east-1.rds.amazonaws.com",
    "port": "5432",
    "dbname": "govtech"
  }'

Restart Application Pods

kubectl rollout restart deployment backend -n govtech
kubectl rollout status deployment backend -n govtech

Verify Connectivity

kubectl logs -f deployment/backend -n govtech | grep "Database connected"

Access Control

Environment-Based IAM Policies

Restrict access to production secrets:

GovTech-Secrets-Read Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SecretsReadDev",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:govtech/dev/*"
    },
    {
      "Sid": "SecretsReadStaging",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:govtech/staging/*"
    },
    {
      "Sid": "DenyProdSecrets",
      "Effect": "Deny",
      "Action": ["secretsmanager:*"],
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:govtech/prod/*"
    }
  ]
}

Developers can access dev/staging secrets but production is explicitly denied.

Kubernetes RBAC for Secrets

rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: govtech-backend-role
  namespace: govtech
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["govtech-secrets"]  # Only specific secret

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: govtech-backend-binding
  namespace: govtech
subjects:
  - kind: ServiceAccount
    name: govtech-backend
    namespace: govtech
roleRef:
  kind: Role
  name: govtech-backend-role
  apiGroup: rbac.authorization.k8s.io

Audit and Monitoring

CloudTrail Logs

All secret access is logged in CloudTrail:

# View recent secret access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::SecretsManager::Secret \
  --max-results 20

# Check who accessed a specific secret
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=govtech/prod/db-credentials

CloudWatch Alarms

Create alerts for suspicious secret access:

# Alert on unauthorized secret access attempts
aws cloudwatch put-metric-alarm \
  --alarm-name govtech-unauthorized-secret-access \
  --alarm-description "Alert on failed secret access attempts" \
  --metric-name UnauthorizedSecretAccess \
  --namespace CloudTrailMetrics \
  --statistic Sum \
  --period 300 \
  --threshold 3 \
  --comparison-operator GreaterThanThreshold

Best Practices

1. Never Commit Secrets to Git

Use .gitignore:

# Kubernetes secrets
kubernetes/secrets.yaml

# Environment files
.env
.env.local
.env.production

# AWS credentials
.aws/credentials

Use templates instead:

cp kubernetes/secrets.yaml.template kubernetes/secrets.yaml
# Edit secrets.yaml with real values (ignored by Git)

2. Rotate Secrets Regularly

Rotation Schedule:

Database passwords: Every 90 days
API keys: Every 90 days
JWT secrets: Every 180 days (requires coordinated deployment)
Access keys: Every 90 days (or use temporary credentials)

Automate rotation:

aws secretsmanager rotate-secret \
  --secret-id govtech/prod/db-credentials \
  --rotation-rules AutomaticallyAfterDays=90

3. Use KMS Encryption

Always encrypt secrets with KMS:

aws secretsmanager create-secret \
  --name govtech/prod/api-key \
  --kms-key-id alias/govtech-prod \
  --secret-string "secret-value"

Benefits:

Centralized key management
Audit trail for key usage
Automatic key rotation
Fine-grained access control

4. Minimize Secret Scope

Principle of least privilege for secrets:

Create separate secrets for each service
Limit secret permissions to specific IAM roles
Use resource tags for access control

Example:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "secretsmanager:ResourceTag/Service": "backend"
    }
  }
}

Troubleshooting

Pod Cannot Access Secrets Manager

Error: AccessDenied: User is not authorized to perform secretsmanager:GetSecretValueDiagnosis:

# Check if IRSA is configured
kubectl describe sa govtech-backend -n govtech | grep Annotations

# Verify IAM role exists
aws iam get-role --role-name govtech-backend-role

# Test from pod
kubectl exec -it <backend-pod> -n govtech -- env | grep AWS

Solution:

Verify ServiceAccount has IAM role annotation
Check IAM role trust policy allows OIDC provider
Ensure IAM policy allows secretsmanager:GetSecretValue

KMS Decryption Fails

Error: AccessDeniedException: User is not authorized to perform kms:DecryptCause: Secret is encrypted with KMS, but IAM role lacks decrypt permission.Solution: Add KMS permissions to IAM role:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:us-east-1:835960996869:key/12345678-1234-1234-1234-123456789012"
}

Get Started

Architecture

Deployment

Multi-Cloud

Security

Operations

Overview

Secret Storage Architecture

AWS Secrets Manager

Secret Naming Convention

Creating Secrets

Updating Secrets

Retrieving Secrets

IRSA (IAM Roles for Service Accounts)

How IRSA Works

Backend Code with IRSA

Kubernetes Secrets (Fallback)

Creating Kubernetes Secrets

Using Secrets in Pods

Secret Rotation

Automatic Rotation with Lambda

Manual Rotation Process

Access Control

Environment-Based IAM Policies

Kubernetes RBAC for Secrets

Audit and Monitoring

CloudTrail Logs

CloudWatch Alarms

Best Practices

Troubleshooting

Next Steps

IAM Policies

Compliance

Build docs developers (and LLMs) love

Get Started

Architecture

Deployment

Multi-Cloud

Security

Operations

​Overview

​Secret Storage Architecture

​AWS Secrets Manager

​Secret Naming Convention

​Creating Secrets

​Updating Secrets

​Retrieving Secrets

​IRSA (IAM Roles for Service Accounts)

​How IRSA Works

​Backend Code with IRSA

​Kubernetes Secrets (Fallback)

​Creating Kubernetes Secrets

​Using Secrets in Pods

​Secret Rotation

​Automatic Rotation with Lambda

​Manual Rotation Process

​Access Control

​Environment-Based IAM Policies

​Kubernetes RBAC for Secrets

​Audit and Monitoring

​CloudTrail Logs

​CloudWatch Alarms

​Best Practices

​Troubleshooting

​Next Steps

IAM Policies

Compliance

Build docs developers (and LLMs) love

Overview

Secret Storage Architecture

AWS Secrets Manager

Secret Naming Convention

Creating Secrets

Updating Secrets

Retrieving Secrets

IRSA (IAM Roles for Service Accounts)

How IRSA Works

Backend Code with IRSA

Kubernetes Secrets (Fallback)

Creating Kubernetes Secrets

Using Secrets in Pods

Secret Rotation

Automatic Rotation with Lambda

Manual Rotation Process

Access Control

Environment-Based IAM Policies

Kubernetes RBAC for Secrets

Audit and Monitoring

CloudTrail Logs

CloudWatch Alarms

Best Practices

Troubleshooting

Next Steps