Security Best Practices

Overview

Operating a cryptocurrency exchange requires implementing defense-in-depth security strategies. This guide covers essential security practices for protecting your Kubernetes infrastructure, data, and services in production.

Infrastructure Security

Network Isolation

Network Policies

Implement network policies to control traffic between pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: exchange-router
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: ingress-nginx
    ports:
    - protocol: TCP
      port: 80
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgresql
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - podSelector:
        matchLabels:
          app: redis
    ports:
    - protocol: TCP
      port: 6379

Key principles:

Default deny all traffic, explicitly allow required connections
Isolate database and Redis to only accept connections from application pods
Restrict egress traffic to prevent data exfiltration
Separate sensitive services into different namespaces

Private Subnets

Deploy databases and internal services in private subnets
Use Cloud NAT for outbound internet access
Restrict access to GKE control plane
Use VPC peering for multi-cluster setups

Cluster Security

GKE Security Hardening

# Enable security features during cluster creation
gcloud container clusters create exchange-cluster \
  --enable-shielded-nodes \
  --enable-autorepair \
  --enable-autoupgrade \
  --enable-stackdriver-kubernetes \
  --enable-network-policy \
  --workload-pool=PROJECT_ID.svc.id.goog \
  --shielded-secure-boot \
  --shielded-integrity-monitoring

Critical features:

Shielded GKE nodes: Protection against rootkits and bootkits
Workload Identity: Secure authentication to Google Cloud services
Binary Authorization: Enforce deployment of trusted container images only
Automatic node upgrades: Keep nodes patched with latest security fixes

Control Plane Access

Restrict access to the Kubernetes API server:

# Authorize specific IP ranges only
gcloud container clusters update exchange-cluster \
  --enable-master-authorized-networks \
  --master-authorized-networks=OFFICE_IP/32,VPN_IP/32

Identity and Access Management

RBAC Configuration

Implement Role-Based Access Control with least privilege:

Read-Only Developer Access

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer-read
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-read-binding
  namespace: default
subjects:
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-read
  apiGroup: rbac.authorization.k8s.io

CI/CD Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: argocd-deployer
  namespace: argocd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-deployer-role
rules:
- apiGroups: ["apps", ""]
  resources: ["deployments", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-deployer-binding
subjects:
- kind: ServiceAccount
  name: argocd-deployer
  namespace: argocd
roleRef:
  kind: ClusterRole
  name: argocd-deployer-role
  apiGroup: rbac.authorization.k8s.io

Workload Identity

Use Workload Identity to authenticate pods with Google Cloud services:

# Enable Workload Identity on cluster
gcloud container clusters update exchange-cluster \
  --workload-pool=PROJECT_ID.svc.id.goog

# Create Google Service Account
gcloud iam service-accounts create exchange-backend-sa

# Bind Kubernetes SA to Google SA
gcloud iam service-accounts add-iam-policy-binding \
  exchange-backend-sa@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[default/exchange-backend]"

# Annotate Kubernetes ServiceAccount
kubectl annotate serviceaccount exchange-backend \
  iam.gke.io/gcp-service-account=exchange-backend-sa@PROJECT_ID.iam.gserviceaccount.com

Secret Management

Sealed Secrets

Sealed Secrets Guide

Comprehensive guide to encrypted secret management

Best practices:

Never commit plain secrets to Git
Use Sealed Secrets for all sensitive configuration
Rotate secrets every 90 days
Use strict scoping (namespace + name binding)
Backup encryption keys securely

Secret Rotation Strategy

# 1. Create new secret version
kubectl create secret generic db-secret-v2 \
  --from-literal=password="new-password" \
  --dry-run=client -o yaml | \
  kubeseal > db-secret-v2-sealed.yml

# 2. Apply new sealed secret
kubectl apply -f db-secret-v2-sealed.yml

# 3. Update deployment to use new secret
kubectl set env deployment/exchange-router \
  DATABASE_PASSWORD=db-secret-v2:password

# 4. Verify rollout
kubectl rollout status deployment/exchange-router

# 5. Delete old secret
kubectl delete secret db-secret-v1

Environment Variable Security

Use Secret references instead of hardcoding:

env:
# Good: Reference from Secret
- name: DATABASE_URL
  valueFrom:
    secretKeyRef:
      name: exchange-router-secret
      key: database_url

# Bad: Hardcoded value
- name: API_KEY
  value: "sk_live_1234567890"  # Never do this!

Pod Security

Pod Security Standards

Apply security contexts to all pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exchange-router
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: router
        image: exchange-router:latest
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        resources:
          limits:
            cpu: "1"
            memory: "512Mi"
          requests:
            cpu: "100m"
            memory: "128Mi"

Key settings:

runAsNonRoot: true: Prevent running as root
readOnlyRootFilesystem: true: Prevent filesystem modifications
allowPrivilegeEscalation: false: Prevent privilege escalation
drop: ALL: Remove all Linux capabilities
Resource limits: Prevent resource exhaustion attacks

Pod Security Admission

Enforce security policies at the namespace level:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Container Image Security

Image Scanning

Scan images for vulnerabilities before deployment:

# Using Trivy
trivy image exchange-router:latest

# Using Google Container Analysis
gcloud container images scan exchange-router:latest

# Fail CI/CD on high/critical vulnerabilities
trivy image --exit-code 1 --severity HIGH,CRITICAL exchange-router:latest

Trusted Image Registry

# Use private registry with image pull secrets
apiVersion: v1
kind: Pod
metadata:
  name: exchange-router
spec:
  imagePullSecrets:
  - name: gcr-json-key
  containers:
  - name: router
    image: gcr.io/PROJECT_ID/exchange-router:v1.2.3
    imagePullPolicy: Always  # Always pull to get latest patches

Image Signing and Verification

Use Binary Authorization to enforce signed images:

# Enable Binary Authorization
gcloud container clusters update exchange-cluster \
  --enable-binauthz

# Create attestation
gcloud beta container binauthz attestations create \
  --artifact-url="gcr.io/PROJECT_ID/exchange-router@sha256:..." \
  --attestor="projects/PROJECT_ID/attestors/prod-attestor"

TLS and Encryption

TLS Certificate Management

TLS Certificates Guide

Automated certificate provisioning with cert-manager

Best practices:

Use TLS 1.3 for all external communications
Implement certificate rotation before expiration
Use strong cipher suites
Enable HSTS headers

Database Encryption

In-transit encryption:

env:
- name: DATABASE_URL
  value: "postgresql://user:pass@host:5432/db?sslmode=require"

At-rest encryption:

Enable encryption on PostgreSQL volumes
Use GKE encryption-at-rest for persistent disks
Use Cloud SQL with customer-managed encryption keys (CMEK)

Monitoring and Auditing

Audit Logging

Enable Kubernetes audit logs:

# Enable audit logging on GKE
gcloud container clusters update exchange-cluster \
  --enable-cloud-logging \
  --logging=SYSTEM,WORKLOAD,API

Monitor for:

Unauthorized API access attempts
Secret access events
Pod exec/attach operations
Configuration changes
Failed authentication attempts

Security Monitoring

Implement monitoring for security events:

# Alert on suspicious pod behavior
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-alerts
data:
  alerts.yml: |
    groups:
    - name: security
      rules:
      - alert: PodExecDetected
        expr: rate(apiserver_audit_event_total{verb="exec"}[5m]) > 0
        annotations:
          summary: "Pod exec command detected"
      - alert: SecretAccessSpike
        expr: rate(apiserver_audit_event_total{objectRef_resource="secrets"}[5m]) > 10
        annotations:
          summary: "Unusual secret access pattern"
      - alert: FailedAuthAttempts
        expr: rate(authentication_attempts{result="error"}[5m]) > 5
        annotations:
          summary: "Multiple failed authentication attempts"

Intrusion Detection

Deploy Falco for runtime security monitoring:

# Install Falco using Helm
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace

Falco detects:

Container breakout attempts
Privilege escalation
Unauthorized file access
Suspicious network activity

Compliance and Governance

Security Policies

Implement organization-wide policies using OPA Gatekeeper:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("Missing required labels: %v", [missing])
        }

Regular Security Audits

Monthly tasks:

Review RBAC permissions
Audit user access logs
Check for unused secrets and service accounts
Review network policies
Scan for vulnerable images

Quarterly tasks:

Rotate all secrets and certificates
Penetration testing
Compliance audit (PCI-DSS, SOC 2)
Review incident response procedures
Update disaster recovery plan

Incident Response

Response Playbook

1. Detection:

Monitor alerts from Falco, Prometheus, GKE
Review audit logs for anomalies
User reports of suspicious behavior

2. Containment:

# Isolate compromised pod
kubectl label pod compromised-pod-123 quarantine=true

# Apply network policy to block traffic
kubectl apply -f quarantine-network-policy.yml

# Scale down compromised deployment
kubectl scale deployment/compromised-app --replicas=0

3. Investigation:

# Capture pod logs
kubectl logs compromised-pod-123 > incident-logs.txt

# Export pod specification
kubectl get pod compromised-pod-123 -o yaml > pod-spec.yml

# Check audit logs
kubectl logs -n kube-system kube-apiserver-XXX | grep compromised-pod

4. Eradication:

# Delete compromised resources
kubectl delete pod compromised-pod-123

# Rotate all secrets
./rotate-secrets.sh

# Rebuild and redeploy from known good image
kubectl set image deployment/app app=gcr.io/project/app:verified-hash

5. Recovery:

Restore from backups if needed
Verify all services are operational
Monitor for recurring issues

6. Post-Incident:

Document incident timeline
Update security policies
Conduct blameless postmortem
Implement preventive controls

Data Protection

Backup Strategy

# Database backups (daily)
kubectl exec -it postgres-0 -- pg_dump -U postgres exchange > backup.sql
gsutil cp backup.sql gs://exchange-backups/$(date +%Y%m%d)/

# Kubernetes resources backup
kubectl get all --all-namespaces -o yaml > k8s-backup.yml
gsutil cp k8s-backup.yml gs://exchange-backups/k8s/$(date +%Y%m%d)/

# Sealed secrets key backup
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key.yml
# Store in secure vault (HashiCorp Vault, Google Secret Manager)

Data Residency

For cryptocurrency exchanges, ensure:

Data stored in compliant regions
User data encrypted at rest
Transaction logs retained per regulatory requirements
PII data properly anonymized in non-production environments

Dependency Management

Keep Components Updated

# Update GKE cluster
gcloud container clusters upgrade exchange-cluster --master
gcloud container clusters upgrade exchange-cluster --node-pool=default-pool

# Update cert-manager
helm upgrade cert-manager jetstack/cert-manager -n cert-manager

# Update sealed-secrets
helm upgrade sealed-secrets bitnami-labs/sealed-secrets -n kube-system

# Update nginx ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.2/deploy/static/provider/cloud/deploy.yaml

Vulnerability Scanning

Automate scanning in CI/CD:

# GitHub Actions example
- name: Scan Docker image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'exchange-router:${{ github.sha }}'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'

Get Started

Setup & Deployment

Core Components

Services

Operations

Security

​Overview

​Infrastructure Security

​Network Isolation

​Network Policies

​Private Subnets

​Cluster Security

​GKE Security Hardening

​Control Plane Access

​Identity and Access Management

​RBAC Configuration

​Read-Only Developer Access

​CI/CD Service Account

​Workload Identity

​Secret Management

​Sealed Secrets

Sealed Secrets Guide

​Secret Rotation Strategy

​Environment Variable Security

​Pod Security

​Pod Security Standards

​Pod Security Admission

​Container Image Security

​Image Scanning

​Trusted Image Registry

​Image Signing and Verification

​TLS and Encryption

​TLS Certificate Management

TLS Certificates Guide

​Database Encryption

​Monitoring and Auditing

​Audit Logging

​Security Monitoring

​Intrusion Detection

​Compliance and Governance

​Security Policies

​Regular Security Audits

​Incident Response

​Response Playbook

​Data Protection

​Backup Strategy

​Data Residency

​Dependency Management

​Keep Components Updated

​Vulnerability Scanning

​Security Checklist

​Pre-Production

​Production Operations

​Resources

​Next Steps

Monitoring

Troubleshooting

Build docs developers (and LLMs) love