Secret Management

KubeLB provides the SyncSecret CRD to securely propagate secrets from tenant clusters to the management cluster in a controlled manner. This is essential for scenarios like TLS certificates for Ingress or credentials for external services.

SyncSecret Overview

SyncSecret is a wrapper around Kubernetes Secret objects that enables secure, controlled synchronization between tenant clusters and the KubeLB management cluster.

SyncSecret provides the same functionality as Kubernetes Secrets but with controlled propagation across cluster boundaries.

Key Features

Controlled Propagation: Secrets are only synced when explicitly wrapped in a SyncSecret
Namespace Isolation: Secrets are confined to tenant-specific namespaces
Automatic Cleanup: Finalizers ensure secrets are properly deleted when no longer needed
UID-Based Naming: Secrets are stored with randomized names to prevent collisions

Architecture

Synchronization Flow

Tenant Cluster                    Management Cluster
┌──────────────────┐             ┌──────────────────────┐
│  SyncSecret CRD  │             │    SyncSecret CRD    │
│  (Original)      │  ─────────> │    (Propagated)      │
│                  │   CCM        │                      │
│  namespace: app  │  Watches    │  namespace: tenant-x │
│  name: tls-cert  │  & Syncs    │  name: <uid>         │
└──────────────────┘             └──────────────────────┘
                                          │
                                          │ Manager
                                          │ Reconciles
                                          ▼
                                 ┌──────────────────┐
                                 │  Kubernetes      │
                                 │  Secret          │
                                 │                  │
                                 │  namespace: ten… │
                                 │  name: <uid>     │
                                 └──────────────────┘

Component Responsibilities

CCM (Cloud Controller Manager)

Location: Runs in tenant clusters Responsibilities:

Watch for SyncSecret resources in the tenant cluster
Propagate SyncSecret to the management cluster’s tenant namespace
Add origin labels (kubelb.k8c.io/origin-name, kubelb.k8c.io/origin-namespace)
Use UID-based naming to prevent conflicts
Handle cleanup when the original SyncSecret is deleted

Reference: internal/controllers/ccm/sync_secret_controller.go:111

Manager

Location: Runs in the management cluster Responsibilities:

Watch for SyncSecret resources in tenant namespaces
Create corresponding Kubernetes Secret with the same data
Maintain owner references for automatic cleanup
Update secrets when SyncSecret changes
Delete secrets when SyncSecret is removed

Reference: internal/controllers/kubelb/sync_secret_controller.go:108

SyncSecret CRD

Resource Definition

apiVersion: kubelb.k8c.io/v1alpha1
kind: SyncSecret
metadata:
  name: tls-certificate
  namespace: application
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-certificate>
  tls.key: <base64-encoded-key>

Fields

The SyncSecret CRD mirrors the Kubernetes Secret structure:

Field	Type	Description
`data`	`map[string][]byte`	Secret data as base64-encoded strings
`stringData`	`map[string]string`	Secret data as plain strings (encoded on creation)
`type`	`corev1.SecretType`	Type of secret (Opaque, kubernetes.io/tls, etc.)
`immutable`	`*bool`	If true, the secret data cannot be updated

Reference: api/ce/kubelb.k8c.io/v1alpha1/sync_secret_types.go:28

Supported Secret Types

SyncSecret supports all Kubernetes secret types:

Opaque - Default secret type
kubernetes.io/tls - TLS certificates
kubernetes.io/dockerconfigjson - Docker registry credentials
kubernetes.io/basic-auth - Basic authentication credentials
kubernetes.io/ssh-auth - SSH authentication credentials
kubernetes.io/service-account-token - Service account tokens

Usage Examples

TLS Certificate for Ingress

Create a SyncSecret for TLS certificates used by Ingress resources:

apiVersion: kubelb.k8c.io/v1alpha1
kind: SyncSecret
metadata:
  name: app-tls
  namespace: production
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDXTCCAkWgAwIBAgIJAKZ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0B...
    -----END PRIVATE KEY-----

Reference the secret in your Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: production
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls  # References the SyncSecret name
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80

Docker Registry Credentials

Create a SyncSecret for private Docker registry access:

apiVersion: kubelb.k8c.io/v1alpha1
kind: SyncSecret
metadata:
  name: registry-credentials
  namespace: production
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "registry.example.com": {
          "username": "user",
          "password": "password",
          "email": "[email protected]",
          "auth": "dXNlcjpwYXNzd29yZA=="
        }
      }
    }

Opaque Secret

Create a generic secret for application configuration:

apiVersion: kubelb.k8c.io/v1alpha1
kind: SyncSecret
metadata:
  name: app-config
  namespace: production
type: Opaque
stringData:
  database-url: "postgresql://user:pass@host:5432/db"
  api-key: "sk-1234567890abcdef"

Security Considerations

UID-Based Naming

Secrets are stored in the management cluster using the SyncSecret’s UID as the name:

// CCM propagates with UID-based name
object.Name = string(object.UID)

// Manager creates Secret with same UID-based name
secret.Name = string(object.UID)

This provides:

Collision Prevention: No risk of name conflicts between tenants
Security Through Obscurity: Secret names are not predictable
Automatic Uniqueness: UIDs are guaranteed unique by Kubernetes

Reference: internal/controllers/ccm/sync_secret_controller.go:120

Origin Tracking

The CCM adds labels to track the original secret location:

metadata:
  labels:
    kubelb.k8c.io/origin-name: tls-certificate
    kubelb.k8c.io/origin-namespace: application

This enables:

Audit trails for secret propagation
Troubleshooting secret sources
Preventing unauthorized modifications

Reference: internal/controllers/ccm/sync_secret_controller.go:115

Owner References

Secrets include owner references to their parent SyncSecret:

metadata:
  ownerReferences:
  - apiVersion: kubelb.k8c.io/v1alpha1
    kind: SyncSecret
    name: <uid>
    uid: <syncsecret-uid>

Benefits:

Automatic Cleanup: Secrets are deleted when the SyncSecret is removed
Garbage Collection: Kubernetes automatically handles orphaned secrets
Clear Ownership: Easy to trace which SyncSecret owns each secret

Reference: internal/controllers/kubelb/sync_secret_controller.go:121

Finalizers

Both CCM and Manager controllers use finalizers (kubelb.k8c.io/lb-finalizer) to ensure proper cleanup: CCM Cleanup:

Delete the SyncSecret from the management cluster
Wait for deletion to complete
Remove finalizer from the tenant cluster SyncSecret

Manager Cleanup:

Delete the Kubernetes Secret from the management cluster
Wait for deletion to complete
Remove finalizer from the SyncSecret

Do not manually remove finalizers. This can cause orphaned secrets in the management cluster.

RBAC Permissions

Manager Permissions

The KubeLB manager requires permissions for both SyncSecrets and Secrets:

rules:
- apiGroups:
  - kubelb.k8c.io
  resources:
  - syncsecrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Full configuration: config/rbac/role.yaml:250

CCM Permissions

The CCM requires permissions to manage SyncSecrets:

rules:
- apiGroups:
  - kubelb.k8c.io
  resources:
  - syncsecrets
  verbs:
  - create
  - update
  - get
  - list
  - delete
  - patch
  - watch

- apiGroups:
  - kubelb.k8c.io
  resources:
  - syncsecrets/status
  verbs:
  - update
  - get
  - patch

CCM does not need direct access to Kubernetes Secrets in the management cluster. All synchronization happens through the SyncSecret CRD.

Reconciliation Logic

CCM Reconciliation

The CCM controller:

Watches SyncSecrets in the tenant cluster
Adds origin labels for tracking
Updates the namespace to the tenant’s management namespace
Resets the UID to generate a new one in the management cluster
Creates or updates the SyncSecret in the management cluster
Handles conflicts with retry logic

Reference: internal/controllers/ccm/sync_secret_controller.go:111

Manager Reconciliation

The Manager controller:

Watches SyncSecrets in tenant namespaces
Creates a Kubernetes Secret with the same data
Uses the SyncSecret UID as the Secret name
Sets owner references for automatic cleanup
Updates the Secret when the SyncSecret changes
Performs semantic equality checks to avoid unnecessary updates

Reference: internal/controllers/kubelb/sync_secret_controller.go:108

Update Detection

Both controllers use semantic equality checks to avoid unnecessary updates:

if equality.Semantic.DeepEqual(existingObj.Data, obj.Data) &&
    equality.Semantic.DeepEqual(existingObj.StringData, obj.StringData) &&
    equality.Semantic.DeepEqual(existingObj.Type, obj.Type) &&
    equality.Semantic.DeepEqual(existingObj.Labels, obj.Labels) &&
    equality.Semantic.DeepEqual(existingObj.Annotations, obj.Annotations) {
    return nil // No update needed
}

This prevents unnecessary API calls and reduces reconciliation load. Reference: internal/controllers/kubelb/sync_secret_controller.go:149

Monitoring and Metrics

Prometheus Metrics

SyncSecret controllers expose Prometheus metrics: CCM Metrics:

kubelb_ccm_sync_secret_reconcile_total{namespace, result} - Total reconciliations
kubelb_ccm_sync_secret_reconcile_duration_seconds{namespace} - Reconciliation duration

Manager Metrics:

kubelb_manager_sync_secret_reconcile_total{namespace, result} - Total reconciliations
kubelb_manager_sync_secret_reconcile_duration_seconds{namespace} - Reconciliation duration

Logging

Both controllers log key events:

SyncSecret creation
SyncSecret updates
Secret synchronization
Cleanup operations
Reconciliation errors

Use structured logging to filter by namespace:

# View SyncSecret operations for a specific tenant
kubectl logs -n kubelb deployment/kubelb-manager | grep "namespace=tenant-production"

Troubleshooting

Secret Not Appearing in Management Cluster

Check:

Verify the SyncSecret exists in the tenant cluster
Check CCM logs for propagation errors
Verify CCM has permissions to create SyncSecrets in the management cluster
Check for network connectivity between clusters

# In tenant cluster
kubectl get syncsecret tls-certificate -n production

# In management cluster
kubectl get syncsecret -n tenant-production

# Check CCM logs
kubectl logs -n kubelb-ccm deployment/kubelb-ccm

Secret Not Created from SyncSecret

Check:

Verify the SyncSecret exists in the management cluster tenant namespace
Check Manager logs for reconciliation errors
Verify Manager has permissions to create Secrets
Check for finalizer issues

# In management cluster
kubectl get syncsecret -n tenant-production

# Check for corresponding secret (using UID)
kubectl get syncsecret <name> -n tenant-production -o jsonpath='{.metadata.uid}'
kubectl get secret <uid> -n tenant-production

# Check Manager logs
kubectl logs -n kubelb deployment/kubelb-manager

Secret Not Updating

Check:

Verify the SyncSecret data has actually changed
Check for reconciliation errors in controller logs
Verify the Secret is not marked as immutable
Check ResourceVersion conflicts

# Compare SyncSecret and Secret data
kubectl get syncsecret <name> -n tenant-production -o yaml
kubectl get secret <uid> -n tenant-production -o yaml

Orphaned Secrets

Check:

Verify SyncSecret has proper finalizers
Check for controller errors during cleanup
Manually delete orphaned secrets if necessary

# Check for secrets without owner references
kubectl get secrets -n tenant-production -o json | jq '.items[] | select(.metadata.ownerReferences == null)'

# Manually delete orphaned secret
kubectl delete secret <uid> -n tenant-production

Manually deleting secrets should be a last resort. Investigate why cleanup failed before removing resources.

Best Practices

Secret Hygiene

Use Immutable Secrets: For secrets that should never change (like TLS certificates), set immutable: true
Rotate Regularly: Update SyncSecrets periodically to rotate credentials
Limit Scope: Only sync secrets that are actually needed in the management cluster
Use Descriptive Names: Name SyncSecrets clearly to indicate their purpose

Access Control

Restrict Creation: Limit who can create SyncSecrets in tenant clusters
Monitor Usage: Track SyncSecret creation and updates
Audit Access: Review who has access to secrets in the management cluster
Use RBAC: Apply proper RBAC policies to control secret access

Operational

Document Secret Purpose: Add annotations describing what each secret is for
Monitor Metrics: Track reconciliation failures and durations
Test Cleanup: Verify secrets are properly deleted when SyncSecrets are removed
Backup Important Secrets: Keep secure backups of critical secrets outside the cluster

Example: TLS Certificate Workflow

Complete workflow for syncing TLS certificates:

1. Create SyncSecret in Tenant Cluster

# Create from files
kubectl create secret tls app-tls \
  --cert=tls.crt \
  --key=tls.key \
  -n production \
  --dry-run=client -o yaml | \
  kubectl create -f - && \
  kubectl apply -f - <<EOF
apiVersion: kubelb.k8c.io/v1alpha1
kind: SyncSecret
metadata:
  name: app-tls
  namespace: production
type: kubernetes.io/tls
data:
  tls.crt: $(base64 -w0 < tls.crt)
  tls.key: $(base64 -w0 < tls.key)
EOF

2. Verify Propagation

# Check in management cluster
kubectl get syncsecret -n tenant-production
kubectl get secret -n tenant-production

3. Use in Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  namespace: production
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80

4. Monitor

# Check metrics
curl -s http://localhost:8080/metrics | grep sync_secret

# View logs
kubectl logs -n kubelb deployment/kubelb-manager | grep app-tls

Overview

Installation

Core Concepts

Guides

Operations

Security

​SyncSecret Overview

​Key Features

​Architecture

​Synchronization Flow

​Component Responsibilities

​CCM (Cloud Controller Manager)

​Manager

​SyncSecret CRD

​Resource Definition

​Fields

​Supported Secret Types

​Usage Examples

​TLS Certificate for Ingress

​Docker Registry Credentials

​Opaque Secret

​Security Considerations

​UID-Based Naming

​Origin Tracking

​Owner References

​Finalizers

​RBAC Permissions

​Manager Permissions

​CCM Permissions

​Reconciliation Logic

​CCM Reconciliation

​Manager Reconciliation

​Update Detection

​Monitoring and Metrics

​Prometheus Metrics

​Logging

​Troubleshooting

​Secret Not Appearing in Management Cluster

​Secret Not Created from SyncSecret

​Secret Not Updating

​Orphaned Secrets

​Best Practices

​Secret Hygiene

​Access Control

​Operational

​Example: TLS Certificate Workflow

​1. Create SyncSecret in Tenant Cluster

​2. Verify Propagation

​3. Use in Ingress

​4. Monitor

​Next Steps

Tenant Isolation

Security Overview

Build docs developers (and LLMs) love

SyncSecret Overview

Key Features

Architecture

Synchronization Flow

Component Responsibilities

CCM (Cloud Controller Manager)

Manager

SyncSecret CRD

Resource Definition

Fields

Supported Secret Types

Usage Examples

TLS Certificate for Ingress

Docker Registry Credentials

Opaque Secret

Security Considerations

UID-Based Naming

Origin Tracking

Owner References

Finalizers

RBAC Permissions

Manager Permissions

CCM Permissions

Reconciliation Logic

CCM Reconciliation

Manager Reconciliation

Update Detection

Monitoring and Metrics

Prometheus Metrics

Logging

Troubleshooting

Secret Not Appearing in Management Cluster

Secret Not Created from SyncSecret

Secret Not Updating

Orphaned Secrets

Best Practices

Secret Hygiene

Access Control

Operational

Example: TLS Certificate Workflow

1. Create SyncSecret in Tenant Cluster

2. Verify Propagation

3. Use in Ingress

4. Monitor

Next Steps