Multiple Controllers - Pipelines as Code

Multi-GitHub Application support is a Technology Preview feature.

Pipelines as Code supports running multiple controller instances on the same cluster, enabling integration with different GitHub instances (such as public GitHub and GitHub Enterprise Server) or separate GitHub Apps.

Use Cases

Multiple GitHub instances - Connect to both github.com and GitHub Enterprise Server
Multiple GitHub Apps - Use different GitHub Apps for different organizations
Isolation - Separate production and development GitHub integrations
Multi-tenancy - Provide dedicated controllers for different teams

Architecture

Each GitHub application requires its own controller deployment with:

Controller Deployment

Deployment

Dedicated controller instance running the pipelines-as-code-controller image

Service

Kubernetes Service to expose the controller’s webhook endpoint

Network Exposure

Ingress/Route

Ingress (Kubernetes) or Route (OpenShift) to make webhooks accessible from GitHub

Secret

Contains GitHub App credentials:

private-key - GitHub App private key
application_id - GitHub App ID
webhook_secret - Webhook secret

ConfigMap

Application-specific configuration settings

While each GitHub App requires its own controller, only one watcher component is needed cluster-wide. The watcher monitors all PipelineRuns regardless of which controller created them.

Controller Configuration

Each controller is configured using environment variables:

PAC_CONTROLLER_LABEL

string

required

Unique identifier for this controller instance. Used to distinguish between multiple controllers.Example: github-enterprise, github-public, dev-github

PAC_CONTROLLER_SECRET

string

required

Name of the Secret containing GitHub App credentials.Example: gh-enterprise-secret

PAC_CONTROLLER_CONFIGMAP

string

Name of the ConfigMap with controller-specific settings. Optional if using default settings.Example: gh-enterprise-config

Deployment Methods

You can deploy additional controllers using:

Automated with startpaac (recommended for CI/testing)
Manual with second-controller.py script (recommended for production)
Manual YAML (for complete customization)

Method 1: Automated with startpaac

startpaac automatically installs and configures second controllers in CI mode.

Prerequisites

startpaac installed and configured
GitHub App credentials stored in a directory

Setup Secret Directory

Create a directory with GitHub App credentials:

mkdir -p ~/secrets-second

echo "12345" > ~/secrets-second/github-application-id
echo "-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----" > ~/secrets-second/github-private-key
echo "my-webhook-secret" > ~/secrets-second/webhook.secret

# Optional: For webhook tunneling with smee.io
echo "https://smee.io/your-channel" > ~/secrets-second/smee

Run startpaac

# Export secret directory location
export PAC_SECOND_SECRET_FOLDER=~/secrets-second

# Run startpaac in CI mode
cd startpaac
./startpaac --ci -a

startpaac will:

Generate TLS certificates for the second controller domain
Create Kubernetes secrets from the secret folder
Deploy the second controller with ingress/route
Configure webhook tunneling if smee URL is provided

Method 2: Using second-controller.py Script

The second-controller.py script simplifies deployment by generating all necessary YAML.

Script Location

# Clone the repository if needed
git clone https://github.com/openshift-pipelines/pipelines-as-code.git
cd pipelines-as-code

# Script is located at:
./hack/second-controller.py

Basic Usage

python3 hack/second-controller.py <LABEL> | kubectl apply -f -

Example:

# Deploy controller for GitHub Enterprise with label "ghe"
python3 hack/second-controller.py ghe | kubectl apply -f -

Script Options

Usage: second-controller.py [-h] [--configmap CONFIGMAP]
                            [--ingress-domain INGRESS_DOMAIN]
                            [--secret SECRET]
                            [--controller-image CONTROLLER_IMAGE]
                            [--gosmee-image GOSMEE_IMAGE]
                            [--smee-url SMEE_URL]
                            [--namespace NAMESPACE]
                            [--openshift-route]
                            LABEL

LABEL

string

required

Unique identifier for the controller (positional argument)

--configmap

string

default:"<LABEL>-configmap"

Name of the ConfigMap for controller settings

--secret

string

default:"<LABEL>-secret"

Name of the Secret containing GitHub App credentials

--ingress-domain

string

Create Kubernetes Ingress with this domain (e.g., ghe.example.com)

--openshift-route

boolean

Create OpenShift Route instead of Ingress

--controller-image

string

Controller container image. Use ko for local builds.

--smee-url

string

Deploy Gosmee sidecar for webhook tunneling via smee.io

--namespace

string

default:"pipelines-as-code"

Kubernetes namespace for deployment

Example Scenarios

Kubernetes with Ingress

python3 hack/second-controller.py github-enterprise \
  --ingress-domain "pac-ghe.k8s.example.com" \
  --secret ghe-app-secret \
  --namespace pipelines-as-code | kubectl apply -f -

OpenShift with Route

python3 hack/second-controller.py enterprise \
  --openshift-route \
  --secret enterprise-github-secret \
  --configmap enterprise-config | oc apply -f -

Local Development with Ko

export KO_DOCKER_REPO=quay.io/myusername

ko apply -f <(
  python3 hack/second-controller.py dev \
  --controller-image=ko \
  --namespace pipelines-as-code
)

Webhook Tunneling with Smee.io

For development or when the cluster is not publicly accessible:

python3 hack/second-controller.py test \
  --smee-url https://smee.io/your-unique-channel | kubectl apply -f -

This deploys a Gosmee sidecar that tunnels webhooks from smee.io to the controller.

Method 3: Manual YAML Deployment

Step 1: Create Secret

Create a Secret with GitHub App credentials:

apiVersion: v1
kind: Secret
metadata:
  name: ghe-secret
  namespace: pipelines-as-code
type: Opaque
stringData:
  github-application-id: "123456"
  github-private-key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEA...
    -----END RSA PRIVATE KEY-----
  webhook.secret: "your-webhook-secret-here"

Apply the secret:

kubectl apply -f ghe-secret.yaml

Step 2: Create ConfigMap (Optional)

apiVersion: v1
kind: ConfigMap
metadata:
  name: ghe-config
  namespace: pipelines-as-code
data:
  application-name: "GHE Pipelines as Code"
  # Add any other controller-specific settings

Step 3: Create Controller Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pac-controller-ghe
  namespace: pipelines-as-code
  labels:
    app.kubernetes.io/name: controller
    app.kubernetes.io/instance: ghe
    app.kubernetes.io/part-of: pipelines-as-code
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: controller
      app.kubernetes.io/instance: ghe
  template:
    metadata:
      labels:
        app.kubernetes.io/name: controller
        app.kubernetes.io/instance: ghe
    spec:
      serviceAccountName: pipelines-as-code-controller
      containers:
        - name: pac-controller
          image: ghcr.io/openshift-pipelines/pipelines-as-code-controller:stable
          env:
            - name: PAC_CONTROLLER_LABEL
              value: "ghe"
            - name: PAC_CONTROLLER_SECRET
              value: "ghe-secret"
            - name: PAC_CONTROLLER_CONFIGMAP
              value: "ghe-config"
          ports:
            - containerPort: 8080
              name: http

Step 4: Create Service

apiVersion: v1
kind: Service
metadata:
  name: pac-controller-ghe
  namespace: pipelines-as-code
spec:
  selector:
    app.kubernetes.io/name: controller
    app.kubernetes.io/instance: ghe
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Step 5: Create Ingress or Route

Kubernetes Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pac-controller-ghe
  namespace: pipelines-as-code
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - pac-ghe.example.com
      secretName: pac-ghe-tls
  rules:
    - host: pac-ghe.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pac-controller-ghe
                port:
                  number: 8080

OpenShift Route:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: pac-controller-ghe
  namespace: pipelines-as-code
spec:
  to:
    kind: Service
    name: pac-controller-ghe
  port:
    targetPort: http
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

Environment Variable Customization

The script respects these environment variables:

export PAC_CONTROLLER_LABEL="my-ghe"              # Controller identifier
export PAC_CONTROLLER_TARGET_NS="pipelines-as-code" # Target namespace
export PAC_CONTROLLER_SECRET="my-ghe-secret"      # Secret name
export PAC_CONTROLLER_CONFIGMAP="my-ghe-config"   # ConfigMap name
export PAC_CONTROLLER_SMEE_URL="https://smee.io/channel" # Smee URL
export PAC_CONTROLLER_IMAGE="ghcr.io/..."         # Controller image

python3 hack/second-controller.py my-ghe | kubectl apply -f -

Verifying Deployment

Check Controller Deployment

# List all controller deployments
kubectl get deployments -n pipelines-as-code -l app.kubernetes.io/name=controller

# Check specific controller status
kubectl get deployment pac-controller-ghe -n pipelines-as-code

# View controller logs
kubectl logs -n pipelines-as-code deployment/pac-controller-ghe -f

Check Service and Ingress

# Verify service
kubectl get svc -n pipelines-as-code

# Check ingress
kubectl get ingress -n pipelines-as-code

# Or route on OpenShift
oc get route -n pipelines-as-code

Test Webhook Endpoint

# Get the webhook URL
kubectl get ingress pac-controller-ghe -n pipelines-as-code -o jsonpath='{.spec.rules[0].host}'

# Test endpoint
curl https://pac-ghe.example.com/

Configuring GitHub App

After deploying the controller, configure your GitHub App:

Set Webhook URL - Use the Ingress/Route URL:
```
https://pac-ghe.example.com
```
Set Webhook Secret - Match the value in your Secret
Configure Permissions - Same as primary GitHub App
Subscribe to Events - Same as primary GitHub App

Repository Assignment

Repositories automatically use the correct controller based on the GitHub App installation. No manual configuration is needed in the Repository CR.

When a webhook arrives, the controller that owns the GitHub App processes it. Multiple controllers can coexist without conflict.

Troubleshooting

Controller Not Starting

Check environment variables:

kubectl get deployment pac-controller-ghe -n pipelines-as-code -o yaml | grep -A5 env

Verify secret exists:

kubectl get secret ghe-secret -n pipelines-as-code

Check controller logs:

kubectl logs -n pipelines-as-code deployment/pac-controller-ghe

Webhooks Not Received

Verify ingress/route is accessible:

curl -v https://pac-ghe.example.com/

Check GitHub App webhook deliveries in GitHub UI Verify webhook secret matches:

kubectl get secret ghe-secret -n pipelines-as-code -o jsonpath='{.data.webhook\.secret}' | base64 -d

Wrong Controller Processing Events

Verify PAC_CONTROLLER_LABEL is unique:

kubectl get deployments -n pipelines-as-code \
  -l app.kubernetes.io/name=controller \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].env[?(@.name=="PAC_CONTROLLER_LABEL")].value}{"\n"}{end}'

Ensure each controller uses a different GitHub App ID:

kubectl get secret ghe-secret -n pipelines-as-code -o jsonpath='{.data.github-application-id}' | base64 -d

Best Practices

Multi-Controller Setup:

Use descriptive labels (github-enterprise, not controller2)
Keep one watcher instance for all controllers
Use separate secrets for each GitHub App
Document which controller serves which GitHub instance
Monitor all controllers with unified metrics
Use consistent naming: pac-controller-<label>

Do not reuse the same GitHub App across multiple controllers. Each controller needs its own GitHub App installation.

Scaling Considerations

Single Watcher - Only one watcher is needed regardless of controller count
Controller Replicas - Each controller can be scaled independently
Resource Limits - Set appropriate resource requests/limits per controller
Namespace Isolation - Controllers can run in separate namespaces if needed

Getting Started

Installation

Git Provider Setup

Core Guides

Advanced Features

Operations

​Use Cases

​Architecture

​Controller Configuration

​Deployment Methods

​Method 1: Automated with startpaac

​Prerequisites

​Setup Secret Directory

​Run startpaac

​Method 2: Using second-controller.py Script

​Script Location

​Basic Usage

​Script Options

​Example Scenarios

​Kubernetes with Ingress

​OpenShift with Route

​Local Development with Ko

​Webhook Tunneling with Smee.io

​Method 3: Manual YAML Deployment

​Step 1: Create Secret

​Step 2: Create ConfigMap (Optional)

​Step 3: Create Controller Deployment

​Step 4: Create Service

​Step 5: Create Ingress or Route

​Environment Variable Customization

​Verifying Deployment

​Check Controller Deployment

​Check Service and Ingress

​Test Webhook Endpoint

​Configuring GitHub App

​Repository Assignment

​Troubleshooting

​Controller Not Starting

​Webhooks Not Received

​Wrong Controller Processing Events

​Best Practices

​Scaling Considerations

​See Also

Build docs developers (and LLMs) love

Use Cases

Architecture

Controller Configuration

Deployment Methods

Method 1: Automated with startpaac

Prerequisites

Setup Secret Directory

Run startpaac

Method 2: Using second-controller.py Script

Script Location

Basic Usage

Script Options

Example Scenarios

Kubernetes with Ingress

OpenShift with Route

Local Development with Ko

Webhook Tunneling with Smee.io

Method 3: Manual YAML Deployment

Step 1: Create Secret

Step 2: Create ConfigMap (Optional)

Step 3: Create Controller Deployment

Step 4: Create Service

Step 5: Create Ingress or Route

Environment Variable Customization

Verifying Deployment

Check Controller Deployment

Check Service and Ingress

Test Webhook Endpoint

Configuring GitHub App

Repository Assignment

Troubleshooting

Controller Not Starting

Webhooks Not Received

Wrong Controller Processing Events

Best Practices

Scaling Considerations

See Also