The KubeLB CCM (Cloud Controller Manager) is deployed in tenant clusters and is responsible for:
Watching Kubernetes Services, Ingresses, and Gateway API resources
Propagating load balancer configurations to the management cluster
Managing node endpoints for load balancing
Synchronizing secrets and certificates when needed
Prerequisites
Before installing the CCM, ensure you have:
A Kubernetes cluster meeting the tenant cluster requirements
KubeLB Manager installed in the management cluster
kubectl configured to access the tenant cluster
Helm 3.x installed
A kubeconfig file with access to the management cluster
A unique tenant name (RFC 1123 lowercase format)
Installation Steps
Create the namespace
Create a dedicated namespace for KubeLB CCM: kubectl create namespace kubelb
Create the management cluster secret
Create a Secret containing the kubeconfig for accessing the management cluster: kubectl --namespace kubelb create secret generic kubelb-cluster \
--from-file=kubelb=/path/to/management-cluster-kubeconfig.yaml
The kubeconfig file must be named kubelb within the Secret. The Secret name can be customized using kubelb.clusterSecretName in values.yaml.
Pull the Helm chart
Download the KubeLB CCM Helm chart from the OCI registry: helm pull oci://quay.io/kubermatic/helm-charts/kubelb-ccm --version=v1.3.1 --untardir "." --untar
Apply CRDs
Install the Custom Resource Definitions (CRDs): kubectl apply -f kubelb-ccm/crds/
Create values file
Create a values.yaml file with your configuration. Here’s a minimal example: kubelb :
# REQUIRED: Unique tenant identifier
tenantName : tenant-cluster-1
# Secret containing management cluster kubeconfig
clusterSecretName : kubelb-cluster
# Node address type for routing (ExternalIP or InternalIP)
nodeAddressType : ExternalIP
# Use LoadBalancer class filtering (false = watch all LoadBalancer services)
useLoadBalancerClass : false
# Resource limits
resources :
limits :
cpu : 500m
memory : 512Mi
requests :
cpu : 100m
memory : 128Mi
The tenantName must be unique across all tenant clusters connected to the same management cluster.
Install the Helm chart
Install KubeLB CCM using Helm: helm install kubelb-ccm kubelb-ccm/ \
--namespace kubelb \
-f values.yaml
Configuration Options
Basic Configuration
Minimal Configuration
With LoadBalancer Class
Internal IP Routing
kubelb :
tenantName : my-tenant
clusterSecretName : kubelb-cluster
nodeAddressType : ExternalIP
Gateway API Support
Enable Gateway API controllers for HTTPRoute, GRPCRoute, and Gateway resources:
kubelb :
enableGatewayAPI : true
useGatewayClass : true
# Optionally install and manage Gateway API CRDs
installGatewayAPICRDs : true
gatewayAPICRDsChannel : standard # or 'experimental'
If installGatewayAPICRDs is false, Gateway API CRDs must be installed manually before enabling Gateway API support.
Layer 7 Resource Controllers
Control which Layer 7 controllers are enabled:
kubelb :
# Ingress controller
disableIngressController : false
useIngressClass : true
# Gateway API controllers
disableGatewayController : false
disableHTTPRouteController : false
disableGRPCRouteController : false
Ingress to Gateway Conversion
Automatically convert Ingress resources to Gateway API HTTPRoutes:
kubelb :
ingressConversion :
enabled : true
gatewayName : kubelb
gatewayNamespace : kubelb
gatewayClass : kubelb
# Optional: Filter by ingress class
ingressClass : "nginx"
# Optional: Domain replacement
domainReplace : ".old.example.com"
domainSuffix : ".new.example.com"
# Annotations for the created Gateway
gatewayAnnotations : "cert-manager.io/cluster-issuer=letsencrypt,external-dns.alpha.kubernetes.io/target=lb.example.com"
# Feature flags
propagateExternalDnsAnnotations : true
copyTLSSecrets : true
disableEnvoyGatewayFeatures : false
Set standaloneMode: true to run only the Ingress converter, disabling all other controllers.
Secret Synchronization
Enable automatic secret synchronization for TLS certificates:
kubelb :
enableSecretSynchronizer : true
Secrets labeled with kubelb.k8c.io/managed-by: kubelb will be automatically synchronized to the management cluster.
Resource Filtering
Control which resources the CCM watches:
Watch All Resources
Class-Based Filtering
kubelb :
useLoadBalancerClass : false
useIngressClass : false
useGatewayClass : false
Monitoring
Enable Prometheus metrics:
serviceMonitor :
enabled : true
metrics :
port : 9445
Enable Grafana dashboards:
grafana :
dashboards :
enabled : true
annotations :
grafana_folder : "KubeLB"
Security
Chart Signature Verification
Verify the Helm chart signature before installation:
cosign verify quay.io/kubermatic/helm-charts/kubelb-ccm:v1.3.1 \
--certificate-identity-regexp= "^https://github.com/kubermatic/kubelb/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
Image Signature Verification
Verify the CCM container image:
cosign verify quay.io/kubermatic/kubelb-ccm:v1.3.1 \
--certificate-identity-regexp= "^https://github.com/kubermatic/kubelb/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
Security Context
The CCM runs with restrictive security settings:
podSecurityContext :
runAsNonRoot : true
seccompProfile :
type : RuntimeDefault
securityContext :
allowPrivilegeEscalation : false
capabilities :
drop :
- ALL
runAsUser : 65532
Kubeconfig Secret Security
Ensure the kubeconfig Secret has minimal RBAC permissions in the management cluster. Only grant access to create and manage LoadBalancer and Route CRDs.
Advanced Configuration
High Availability
Run multiple replicas with leader election:
replicaCount : 2
kubelb :
enableLeaderElection : true
Custom Volumes
Mount additional volumes for certificates or configuration:
extraVolumes :
- name : ca-certs
configMap :
name : custom-ca-certs
extraVolumeMounts :
- name : ca-certs
mountPath : /etc/ssl/certs/custom
readOnly : true
Priority Class
Set pod priority for critical workloads:
priorityClassName : "system-cluster-critical"
Logging Configuration
Adjust log verbosity:
kubelb :
logLevel : debug # debug, info, error, panic, or numeric value
Multiple Tenant Clusters
To connect multiple tenant clusters to the same management cluster:
Use unique tenant names
Ensure each tenant cluster has a unique tenantName value: # Cluster 1
kubelb :
tenantName : production-us-east
# Cluster 2
kubelb :
tenantName : production-eu-west
Use the same management cluster
All tenant clusters should use the same management cluster kubeconfig in their secrets.
Install CCM in each cluster
Repeat the installation steps in each tenant cluster with the appropriate tenant name.
Upgrading
To upgrade KubeLB CCM to a new version:
Update CRDs
kubectl apply -f kubelb-ccm/crds/
Upgrade Helm release
helm upgrade kubelb-ccm oci://quay.io/kubermatic/helm-charts/kubelb-ccm \
--version=v1.3.1 \
--namespace kubelb \
-f values.yaml
Troubleshooting
Check CCM Status
kubectl get pods -n kubelb -l app.kubernetes.io/name=kubelb-ccm
View CCM Logs
kubectl logs -n kubelb -l app.kubernetes.io/name=kubelb-ccm -f
Verify Management Cluster Connection
kubectl exec -n kubelb deploy/kubelb-ccm -- \
cat /kubelb/kubelb | grep server
Check LoadBalancer Resources
Verify LoadBalancer CRDs are being created in the management cluster:
# Run on management cluster
kubectl get loadbalancers -A
Common Issues
CCM fails to connect to management cluster : Verify the kubeconfig Secret is correctly created and the management cluster API is accessible from the tenant cluster.
LoadBalancer services not getting external IPs : Check that useLoadBalancerClass is set correctly and services have the appropriate loadBalancerClass annotation if filtering is enabled.
Gateway API controllers failing to start : Ensure Gateway API CRDs are installed or set installGatewayAPICRDs: true.
Next Steps
After installing the CCM:
Verify the installation to ensure the CCM is communicating with the Manager
Create your first LoadBalancer service to test the setup