Secrets

Overview

Kubernetes Secrets provide a secure way to store and manage sensitive information such as passwords, OAuth tokens, and SSH keys. The exchange platform uses both standard Secrets and Sealed Secrets for enhanced security in version control.

Secret Types

Standard Kubernetes Secret

The exchange-router-secret Secret stores sensitive configuration for the backend router service. Location: backend/sample-secret.md

apiVersion: v1
kind: Secret
metadata:
  name: exchange-router-secret
data:
  server_addr: MC4wLjAuMDo4MDgw
  database_url: cG9zdGdyZXM6Ly9yb290OnJvb3RAZXhjaGFuZ2UtcG9zdGdyZXMtc2VydmljZS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjgwL2V4Y2hhbmdlLWRi
  redis_url: cmVkaXM6Ly9leGNoYW5nZS1yZWRpcy1zZXJ2aWNlLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODA=

Decoded Values:

Key	Decoded Value	Description
`server_addr`	`0.0.0.0:8080`	Backend router bind address and port
`database_url`	`postgres://root:root@exchange-postgres-service...`	PostgreSQL connection string
`redis_url`	`redis://exchange-redis-service...`	Redis connection string

Secret data values are base64-encoded, not encrypted. Base64 encoding is not a security measure.

Sealed Secret

Sealed Secrets provide encrypted Secrets that can be safely stored in version control. Location: backend/sealed-secret.yml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: exchange-router-secret
  namespace: default
spec:
  encryptedData:
    database_url: AgAUEdKobv+RYQTmkqi/4zFp38QYoE0mEk8HkU4Ra8U7ZP2HtewG...
    redis_url: AgA7Tzk4yAFU7Lxwv4OlgpJyXxi9hr/7qC4IyxD3bPAkaVPHi/6eS9P...
    server_addr: AgDNCrl5xf/DQLaY/jSYWxaNGHPTwSUOWRChWedYoHRS2ElVoI+p7...
  template:
    metadata:
      creationTimestamp: null
      name: exchange-router-secret
      namespace: default

Key Features:

Encrypted using cluster-specific keys
Safe to commit to Git repositories
Automatically decrypted by the Sealed Secrets controller
Creates a standard Kubernetes Secret when applied

Secret Structure

Metadata

Each Secret must specify:

name: Unique identifier for the Secret
namespace: Kubernetes namespace (typically default)

Data Format

Secret data can be provided in two formats: 1. data field: Base64-encoded values

data:
  username: YWRtaW4=  # "admin" encoded
  password: cGFzc3dvcmQ=  # "password" encoded

2. stringData field: Plain text values (automatically encoded)

stringData:
  username: admin
  password: password

Never commit Secrets with stringData or decoded values to version control. Use Sealed Secrets instead.

Usage in Deployments

Referencing Secrets as Environment Variables

The backend deployment references Secret values using secretKeyRef: Reference: backend/deployment.yml:25-40

env:
  - name: SERVER_ADDR
    valueFrom:
      secretKeyRef:
        name: exchange-router-secret
        key: server_addr
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: exchange-router-secret
        key: database_url
  - name: REDIS_URL
    valueFrom:
      secretKeyRef:
        name: exchange-router-secret
        key: redis_url

Mounting Secrets as Volumes

Alternatively, mount Secrets as files:

volumeMounts:
  - name: secret-volume
    mountPath: /etc/secrets
    readOnly: true
volumes:
  - name: secret-volume
    secret:
      secretName: exchange-router-secret

Sealed Secrets Setup

Installation

1. Install Helm Chart

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system \
  --set-string fullnameOverride=sealed-secrets-controller \
  sealed-secrets/sealed-secrets

2. Install kubeseal CLI

KUBESEAL_VERSION='0.29.0'
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

3. Verify Installation

kubeseal --fetch-cert

Creating Sealed Secrets

Method 1: From Existing Secret

kubeseal --format yaml < sample-secret.yml > sealed-secret.yml

Method 2: Direct Creation

kubectl create secret generic exchange-router-secret \
  --dry-run=client \
  --from-literal=server_addr=0.0.0.0:8080 \
  --from-literal=database_url=postgres://... \
  --from-literal=redis_url=redis://... \
  -o yaml | kubeseal \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  --format yaml > sealed-secret.yml

Method 3: Offline Encryption For environments without cluster access:

# Fetch the public certificate
kubeseal \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  --fetch-cert > mycert.pem

# Create Sealed Secret using local certificate
kubectl create secret generic secret-name \
  --dry-run=client \
  --from-literal=foo=bar \
  -o yaml | kubeseal \
  --cert mycert.pem \
  --format yaml > sealed-secret.yml

Applying Sealed Secrets

# Apply the Sealed Secret
kubectl apply -f sealed-secret.yml

# Verify the Secret was created
kubectl get secret exchange-router-secret

# View decrypted Secret (requires permissions)
kubectl get secret exchange-router-secret -o yaml

Secret Management

Creating Secrets

From Literal Values:

kubectl create secret generic exchange-router-secret \
  --from-literal=server_addr=0.0.0.0:8080 \
  --from-literal=database_url=postgres://user:pass@host:port/db \
  --from-literal=redis_url=redis://host:port

From Files:

kubectl create secret generic exchange-router-secret \
  --from-file=server_addr=./server-addr.txt \
  --from-file=database_url=./db-url.txt

From YAML Manifest:

kubectl apply -f sample-secret.yml

Viewing Secrets

# List all Secrets
kubectl get secrets

# Describe Secret metadata
kubectl describe secret exchange-router-secret

# Get Secret data (base64 encoded)
kubectl get secret exchange-router-secret -o yaml

# Decode specific key
kubectl get secret exchange-router-secret -o jsonpath='{.data.server_addr}' | base64 -d

Updating Secrets

# Edit Secret directly
kubectl edit secret exchange-router-secret

# Replace from file
kubectl apply -f sample-secret.yml

# Update specific key
kubectl patch secret exchange-router-secret -p '{"data":{"server_addr":"'$(echo -n "0.0.0.0:9090" | base64)'"}}' 

After updating a Secret, restart pods to load new values:

kubectl rollout restart deployment exchange-router-deployment

Deleting Secrets

kubectl delete secret exchange-router-secret

Best Practices

Security

Never Commit Plain Secrets
- Use Sealed Secrets for version control
- Add *secret*.yml to .gitignore (except sealed-secret.yml)
- Rotate credentials regularly
Principle of Least Privilege
- Limit Secret access using RBAC
- Use separate Secrets for different services
- Avoid mounting Secrets in unnecessary containers
Encryption at Rest
- Enable etcd encryption in production clusters
- Use cloud provider KMS integration when available
External Secret Management
- Consider integrating with HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault
- Use the External Secrets Operator for production environments

Naming Conventions

Follow the pattern: <service>-secret
Example: exchange-router-secret, exchange-api-secret
Use consistent naming across ConfigMaps and Secrets

Documentation

Document all Secret keys and their purposes
Maintain a separate secure document listing Secret values for disaster recovery
Track Secret rotation schedules

Migration from ConfigMaps

The exchange-postgres-config ConfigMap currently stores database passwords. These should be migrated to Secrets:

# Create a new Secret for PostgreSQL
apiVersion: v1
kind: Secret
metadata:
  name: exchange-postgres-secret
stringData:
  POSTGRES_PASSWORD: root

Update deployment to reference both:

env:
  - name: POSTGRES_USER
    valueFrom:
      configMapKeyRef:
        name: exchange-postgres-config
        key: POSTGRES_USER
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: exchange-postgres-secret
        key: POSTGRES_PASSWORD

Troubleshooting

Secret Not Found

# Check if Secret exists
kubectl get secret exchange-router-secret

# Check namespace
kubectl get secret exchange-router-secret -n default

Sealed Secret Not Decrypting

# Check Sealed Secrets controller status
kubectl get pods -n kube-system | grep sealed-secrets

# View controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller

# Verify SealedSecret resource
kubectl get sealedsecret exchange-router-secret

Permission Denied

# Check RBAC permissions
kubectl auth can-i get secrets
kubectl auth can-i create secrets

Verify Secret Values in Pods

# Get pod name
kubectl get pods -l app=exchange-router

# Check environment variables
kubectl exec <pod-name> -- env | grep -E '(SERVER_ADDR|DATABASE_URL|REDIS_URL)'

# Note: Avoid printing sensitive values in production

Reference Documentation

Official Resources:

Related Pages:

ConfigMaps - Non-sensitive configuration
Environment Variables - Complete variable reference

Kubernetes Manifests

Configuration

Overview

Secret Types

Standard Kubernetes Secret

Sealed Secret

Secret Structure

Metadata

Data Format

Usage in Deployments

Referencing Secrets as Environment Variables

Mounting Secrets as Volumes

Sealed Secrets Setup

Installation

Creating Sealed Secrets

Applying Sealed Secrets

Secret Management

Creating Secrets

Viewing Secrets

Updating Secrets

Deleting Secrets

Best Practices

Security

Naming Conventions

Documentation

Migration from ConfigMaps

Troubleshooting

Secret Not Found

Sealed Secret Not Decrypting

Permission Denied

Verify Secret Values in Pods

Reference Documentation

Build docs developers (and LLMs) love

Kubernetes Manifests

Configuration

​Overview

​Secret Types

​Standard Kubernetes Secret

​Sealed Secret

​Secret Structure

​Metadata

​Data Format

​Usage in Deployments

​Referencing Secrets as Environment Variables

​Mounting Secrets as Volumes

​Sealed Secrets Setup

​Installation

​Creating Sealed Secrets

​Applying Sealed Secrets

​Secret Management

​Creating Secrets

​Viewing Secrets

​Updating Secrets

​Deleting Secrets

​Best Practices

​Security

​Naming Conventions

​Documentation

​Migration from ConfigMaps

​Troubleshooting

​Secret Not Found

​Sealed Secret Not Decrypting

​Permission Denied

​Verify Secret Values in Pods

​Reference Documentation

Build docs developers (and LLMs) love

Overview

Secret Types

Standard Kubernetes Secret

Sealed Secret

Secret Structure

Metadata

Data Format

Usage in Deployments

Referencing Secrets as Environment Variables

Mounting Secrets as Volumes

Sealed Secrets Setup

Installation

Creating Sealed Secrets

Applying Sealed Secrets

Secret Management

Creating Secrets

Viewing Secrets

Updating Secrets

Deleting Secrets

Best Practices

Security

Naming Conventions

Documentation

Migration from ConfigMaps

Troubleshooting

Secret Not Found

Sealed Secret Not Decrypting

Permission Denied

Verify Secret Values in Pods

Reference Documentation