Secrets Management

Overview

The K8s Scheduler implements a sophisticated three-tier secrets management system that integrates with HashiCorp Vault and uses External Secrets Operator (ESO) to inject secrets into Kubernetes pods.

Secrets Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        VAULT                                     │
│                                                                  │
│  users/{userId}/                                                 │
│  ├── secrets/              # User-level (global) secrets         │
│  ├── templates/{name}/     # Template secrets                    │
│  │   └── services/{svc}/   # Per-service template secrets        │
│  └── deployments/{name}/   # Deployment-specific secrets         │
│      └── services/{svc}/   # Per-service deployment secrets      │
└─────────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────┐
│                    EXTERNAL SECRETS OPERATOR                     │
│                                                                  │
│  ExternalSecret CR ──syncs──▶ K8s Secret                        │
│  (references Vault path)      (mounted in pods)                  │
└─────────────────────────────────────────────────────────────────┘

Secret Levels

Secrets are organized into three hierarchical levels:

1. User Secrets (Global)

Path: users/{userId}/secrets Lifecycle: Persistent - survives deployment deletions Use Case: Shared secrets across all deployments Examples:

API keys for external services (OpenAI, Stripe, SendGrid)
Database credentials used by multiple apps
Third-party service tokens
Cloud provider credentials (AWS, GCP, Azure)

Access: Available to all deployments created by the user

User secrets are ideal for credentials that need to be shared across multiple deployments, reducing duplication and making updates easier.

2. Template Secrets

Path: users/{userId}/templates/{templateName} Lifecycle: Persistent - tied to the template Use Case: Template-specific configuration and defaults Examples:

Default database connection strings
Application-specific API keys
Service-specific configuration
Default environment variables

Access: Available to all deployments created from the template

Template secrets provide sensible defaults that can be overridden by deployment-specific secrets.

3. Deployment Secrets

Path: users/{userId}/deployments/{deploymentName} Lifecycle: Ephemeral - deleted when deployment is deleted Use Case: Deployment-specific overrides and configuration Examples:

Environment-specific API keys (dev, staging, prod)
Instance-specific credentials
Deployment-specific feature flags
Override values for template secrets

Access: Only available to the specific deployment

Deployment secrets are deleted when you delete the deployment. Back up critical secrets before deletion.

Secret Precedence

When multiple secret levels define the same key, deployment secrets take precedence:

Deployment Secrets (highest priority)
        ↓
 Template Secrets
        ↓
   User Secrets (lowest priority)

Example: If all three levels define DATABASE_URL:

# User secret
DATABASE_URL=postgres://default-host/db

# Template secret
DATABASE_URL=postgres://template-host/app-db

# Deployment secret
DATABASE_URL=postgres://prod-host/prod-db

# Result: The deployment receives:
DATABASE_URL=postgres://prod-host/prod-db

Per-Service Secrets

For multi-service deployments, secrets can be scoped to specific services: Path Pattern:

users/{userId}/
  ├── secrets/                          # Shared by all services
  ├── templates/{name}/services/{svc}/  # Service-specific template secrets
  └── deployments/{name}/services/{svc}/# Service-specific deployment secrets

Example: For a deployment with frontend and backend services:

# Shared secret (available to both)
users/user_123/secrets:
  - SHARED_API_KEY=abc123

# Frontend-specific
users/user_123/deployments/my-app/services/frontend:
  - NEXT_PUBLIC_API_URL=https://api.example.com

# Backend-specific
users/user_123/deployments/my-app/services/backend:
  - DATABASE_URL=postgres://...
  - REDIS_URL=redis://...

Secrets Backends

The K8s Scheduler supports three secrets backend options:

Vault (Recommended)
AWS Secrets Manager
Database (Development)

HashiCorp Vault

Pros:

Production-grade secrets management
Audit logging and access control
Dynamic secrets support
Encryption at rest and in transit
Integration with External Secrets Operator

Configuration:

SECRETS_BACKEND=vault
VAULT_ADDR=https://vault.example.com
VAULT_TOKEN=hvs.XXXXXX
VAULT_MOUNT_PATH=secret

Requirements:

Running Vault instance
KV v2 secrets engine enabled
External Secrets Operator in cluster

AWS Secrets Manager

Pros:

Fully managed AWS service
Automatic rotation support
IAM-based access control
Integration with AWS services

Configuration:

SECRETS_BACKEND=aws
AWS_REGION=us-east-1
# Uses IAM role or credentials from environment

Requirements:

AWS account with Secrets Manager enabled
IAM permissions for secrets access
External Secrets Operator with AWS provider

PostgreSQL Database

Pros:

Simple setup for development
No additional infrastructure
Encrypted at rest in database

Cons:

Not recommended for production
Limited audit capabilities
No dynamic secrets

Configuration:

SECRETS_BACKEND=database
SECRETS_ENCRYPTION_KEY=base64-encoded-32-byte-key

The database backend is suitable for development only. Use Vault or AWS Secrets Manager for production deployments.

Managing Secrets via API

Creating User Secrets

curl -X PUT https://api.example.com/api/secrets/OPENAI_API_KEY \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "sk-proj-..."
  }'

Listing User Secrets

curl https://api.example.com/api/secrets \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "secrets": [
    {
      "key": "OPENAI_API_KEY",
      "created_at": "2024-01-15T10:30:00Z",
      "updated_at": "2024-01-15T10:30:00Z"
    },
    {
      "key": "DATABASE_URL",
      "created_at": "2024-01-10T09:15:00Z",
      "updated_at": "2024-01-20T14:22:00Z"
    }
  ]
}

Secret values are never returned by the API for security reasons. Only keys and metadata are exposed.

Creating Template Secrets

curl -X PUT https://api.example.com/api/templates/{templateName}/secrets/{key} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "secret-value"
  }'

Creating Deployment Secrets

curl -X PUT https://api.example.com/api/deployments/{deploymentName}/secrets/{key} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "secret-value"
  }'

Creating Service-Specific Secrets

curl -X PUT https://api.example.com/api/deployments/{deploymentName}/services/{serviceName}/secrets/{key} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "value": "service-specific-secret"
  }'

Deleting Secrets

curl -X DELETE https://api.example.com/api/secrets/{key} \
  -H "Authorization: Bearer $TOKEN"

Secrets in Deployments

Automatic Secret Injection

When you create a deployment, the operator automatically:

Creates an ExternalSecret CR for each service
ESO syncs secrets from Vault to a Kubernetes Secret
The Secret is mounted in the pod as environment variables

Generated ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-frontend-secrets
  namespace: sandbox-user_123
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: my-app-frontend-secrets
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: users/user_123/secrets
  - extract:
      key: users/user_123/templates/librechat
  - extract:
      key: users/user_123/templates/librechat/services/frontend
  - extract:
      key: users/user_123/deployments/my-app
  - extract:
      key: users/user_123/deployments/my-app/services/frontend

Pod Environment Variables

Secrets are automatically injected as environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: my-app-frontend
spec:
  containers:
  - name: frontend
    image: ghcr.io/myapp/frontend:latest
    envFrom:
    - secretRef:
        name: my-app-frontend-secrets

Vault Setup

Enabling KV v2 Engine

vault secrets enable -path=secret kv-v2

Creating Vault Policy

# vault-policy.hcl
path "secret/data/users/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/metadata/users/*" {
  capabilities = ["read", "list"]
}

Apply the policy:

vault policy write k8s-scheduler vault-policy.hcl

Kubernetes Auth

Configure Vault to authenticate the External Secrets Operator:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets-system \
  policies=k8s-scheduler \
  ttl=24h

External Secrets Operator Setup

Install ESO

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

ClusterSecretStore Configuration

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets-system"

Apply the configuration:

kubectl apply -f cluster-secret-store.yaml

Verify ESO Setup

# Check ESO pods
kubectl get pods -n external-secrets-system

# Verify ClusterSecretStore
kubectl get clustersecretstore vault-backend

# Check status
kubectl describe clustersecretstore vault-backend

Security Best Practices

Rotate Secrets Regularly

Update secrets periodically, especially after team member departures or security incidents.

Use Per-Service Secrets

Scope secrets to specific services to limit exposure if one service is compromised.

Audit Secret Access

Enable Vault audit logging to track who accesses secrets and when.

Limit Secret Permissions

Use RBAC to control who can view, create, or delete secrets. Viewers cannot access secrets.

Avoid Hardcoding

Never commit secrets to source control. Always use the secrets management system.

Use Strong Encryption

Ensure Vault is configured with proper encryption at rest and in transit (TLS).

Troubleshooting

Secrets Not Appearing in Pods

Check ExternalSecret status:

kubectl get externalsecret -n sandbox-user_123
kubectl describe externalsecret my-app-frontend-secrets -n sandbox-user_123

Common issues:

ClusterSecretStore not configured
Vault authentication failed
Secret path doesn’t exist in Vault
ESO doesn’t have permissions

ClusterSecretStore Not Ready

Verify Vault connectivity:

kubectl logs -n external-secrets-system -l app.kubernetes.io/name=external-secrets

Check Vault auth:

vault read auth/kubernetes/role/external-secrets

Secret Sync Delays

ExternalSecrets refresh every 1 minute by default. To force an immediate sync:

kubectl annotate externalsecret my-app-frontend-secrets \
  force-sync=$(date +%s) \
  -n sandbox-user_123

Database Backend Encryption Key

If using the database backend, generate a secure key:

openssl rand -base64 32

Set as environment variable:

export SECRETS_ENCRYPTION_KEY="your-base64-key-here"

API - Secrets

Secrets management API reference

Templates

Using secrets in templates

Configuration

Secrets backend configuration options

Dependencies

Vault and External Secrets Operator setup

Getting Started

Core Concepts

Components

Deployment

API Reference

​Overview

​Secrets Architecture

​Secret Levels

​1. User Secrets (Global)

​2. Template Secrets

​3. Deployment Secrets

​Secret Precedence

​Per-Service Secrets

​Secrets Backends

​HashiCorp Vault

​AWS Secrets Manager

​PostgreSQL Database

​Managing Secrets via API

​Creating User Secrets

​Listing User Secrets

​Creating Template Secrets

​Creating Deployment Secrets

​Creating Service-Specific Secrets

​Deleting Secrets

​Secrets in Deployments

​Automatic Secret Injection

​Pod Environment Variables

​Vault Setup

​Enabling KV v2 Engine

​Creating Vault Policy

​Kubernetes Auth

​External Secrets Operator Setup

​Install ESO

​ClusterSecretStore Configuration

​Verify ESO Setup

​Security Best Practices

Rotate Secrets Regularly

Use Per-Service Secrets

Audit Secret Access

Limit Secret Permissions

Avoid Hardcoding

Use Strong Encryption

​Troubleshooting

​Secrets Not Appearing in Pods

​ClusterSecretStore Not Ready

​Secret Sync Delays

​Database Backend Encryption Key

​Related Documentation

API - Secrets

Templates

Configuration

Dependencies

Build docs developers (and LLMs) love

Overview

Secrets Architecture

Secret Levels

1. User Secrets (Global)

2. Template Secrets

3. Deployment Secrets

Secret Precedence

Per-Service Secrets

Secrets Backends

HashiCorp Vault

AWS Secrets Manager

PostgreSQL Database

Managing Secrets via API

Creating User Secrets

Listing User Secrets

Creating Template Secrets

Creating Deployment Secrets

Creating Service-Specific Secrets

Deleting Secrets

Secrets in Deployments

Automatic Secret Injection

Pod Environment Variables

Vault Setup

Enabling KV v2 Engine

Creating Vault Policy

Kubernetes Auth

External Secrets Operator Setup

Install ESO

ClusterSecretStore Configuration

Verify ESO Setup

Security Best Practices

Troubleshooting

Secrets Not Appearing in Pods

ClusterSecretStore Not Ready

Secret Sync Delays

Database Backend Encryption Key

Related Documentation