Security - Cadence

Overview

Cadence provides comprehensive security features including OAuth-based authorization, mutual TLS, SASL authentication for Kafka, and network encryption. This guide covers production security best practices.

Authorization

Cadence supports two authorization modes:

NoopAuthorizer - No authorization (development only)
OAuthAuthorizer - JWT-based authorization with RS256

OAuth Authorization

Configuration

authorization:
  oauthAuthorizer:
    enable: true
    maxJwtTTL: 86400  # 24 hours in seconds
    jwtCredentials:
      algorithm: "RS256"
      publicKey: "/etc/cadence/keys/public.pem"

JWT Public Key Configuration

Cadence validates JWTs using RSA public keys. The public key must be in PEM format:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
-----END PUBLIC KEY-----

Only RS256 (RSA with SHA-256) algorithm is supported. Ensure your identity provider issues RS256 JWTs.

Using External Identity Providers

For external OAuth providers (Auth0, Okta, etc.), use JWKS URL:

authorization:
  oauthAuthorizer:
    enable: true
    maxJwtTTL: 86400
    provider:
      jwksURL: "https://your-domain.auth0.com/.well-known/jwks.json"
      groupsAttributePath: "https://your-app/groups"
      adminAttributePath: "https://your-app/admin"

JWT Claims

Cadence expects these claims in JWTs:

{
  "sub": "[email protected]",
  "exp": 1709548800,
  "iat": 1709545200,
  "groups": ["team-a", "team-b"],
  "admin": true
}

sub: Subject (user identifier)
exp: Expiration timestamp (must be within maxJwtTTL)
iat: Issued at timestamp
groups: User group memberships (custom claim path configurable)
admin: Admin flag (custom claim path configurable)

Authorization Validation

Validation rules:

// From common/config/authorization.go

// 1. Only one authorizer can be enabled
if oauthAuthorizer.Enable && noopAuthorizer.Enable {
    return error
}

// 2. MaxJwtTTL must be positive
if maxJwtTTL <= 0 {
    return error
}

// 3. Either jwtCredentials or provider required
if jwtCredentials == nil && provider == nil {
    return error
}

// 4. Algorithm must be RS256
if algorithm != "RS256" {
    return error
}

Cadence validates JWT signatures but does not perform additional authorization checks beyond signature validation. Implement fine-grained access control in your API gateway or via domain-level permissions.

Client Authentication

Clients must include the JWT in RPC headers:

import (
    "go.uber.org/cadence/client"
    "go.uber.org/yarpc"
)

c, err := client.NewClient(
    client.Options{
        HostPort:     "cadence-frontend:7933",
        Domain:       "my-domain",
        MetricsScope: metricsScope,
        HeadersProvider: func() map[string]string {
            return map[string]string{
                "authorization": "Bearer " + jwtToken,
            }
        },
    },
)

TLS/SSL Configuration

Server-Side TLS

Enable TLS for gRPC endpoints:

services:
  frontend:
    rpc:
      grpcPort: 7833
      tls:
        enabled: true
        certFile: "/etc/cadence/tls/server.crt"
        keyFile: "/etc/cadence/tls/server.key"
        caFile: "/etc/cadence/tls/ca.crt"
        requireClientAuth: true
        enableHostVerification: true
        serverName: "cadence-frontend.example.com"

TLS Configuration Options

Option	Description	Required
`enabled`	Enable TLS	Yes
`certFile`	Server certificate path	Yes
`keyFile`	Server private key path	Yes
`caFile`	CA certificate for client verification	For mTLS
`caFiles`	Multiple CA certificates	For mTLS
`requireClientAuth`	Enforce mutual TLS	No
`enableHostVerification`	Verify client hostname	No
`serverName`	Expected server name in certificate	No

Mutual TLS (mTLS)

For mutual TLS, both server and client present certificates:

Server Configuration

services:
  frontend:
    rpc:
      tls:
        enabled: true
        certFile: "/etc/cadence/tls/server.crt"
        keyFile: "/etc/cadence/tls/server.key"
        caFile: "/etc/cadence/tls/ca.crt"
        requireClientAuth: true  # Require client certificates

Client Configuration

import (
    "crypto/tls"
    "crypto/x509"
    "go.uber.org/cadence/client"
)

// Load client certificate
cert, err := tls.LoadX509KeyPair(
    "/etc/cadence/tls/client.crt",
    "/etc/cadence/tls/client.key",
)

// Load CA certificate
caCert, err := ioutil.ReadFile("/etc/cadence/tls/ca.crt")
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)

tlsConfig := &tls.Config{
    Certificates: []tls.Certificate{cert},
    RootCAs:      caCertPool,
    ServerName:   "cadence-frontend.example.com",
}

c, err := client.NewClient(
    client.Options{
        HostPort:  "cadence-frontend:7833",
        Domain:    "my-domain",
        TLSConfig: tlsConfig,
    },
)

Cross-Cluster TLS

For cross-DC replication, configure TLS per cluster:

clusterGroupMetadata:
  currentClusterName: "cluster1"
  clusterGroup:
    cluster1:
      enabled: true
      initialFailoverVersion: 1
      rpcName: "cadence-frontend"
      rpcAddress: "cluster1.example.com:7833"
      rpcTransport: "grpc"
      tls:
        enabled: true
        caFile: "/etc/cadence/tls/cluster1-ca.crt"
        certFile: "/etc/cadence/tls/cluster1-client.crt"
        keyFile: "/etc/cadence/tls/cluster1-client.key"
        enableHostVerification: true
        serverName: "cluster1.example.com"
    cluster2:
      enabled: true
      initialFailoverVersion: 2
      rpcName: "cadence-frontend"
      rpcAddress: "cluster2.example.com:7833"
      rpcTransport: "grpc"
      tls:
        enabled: true
        caFile: "/etc/cadence/tls/cluster2-ca.crt"
        certFile: "/etc/cadence/tls/cluster2-client.crt"
        keyFile: "/etc/cadence/tls/cluster2-client.key"
        enableHostVerification: true
        serverName: "cluster2.example.com"

Database Security

Cassandra TLS

persistence:
  datastores:
    default:
      cassandra:
        hosts: "cassandra1,cassandra2,cassandra3"
        keyspace: "cadence"
        tls:
          enabled: true
          caFile: "/etc/cadence/tls/cassandra-ca.crt"
          certFile: "/etc/cadence/tls/cassandra-client.crt"
          keyFile: "/etc/cadence/tls/cassandra-client.key"
          enableHostVerification: true

MySQL/PostgreSQL TLS

MySQL

persistence:
  datastores:
    default:
      sql:
        pluginName: "mysql"
        connectAddr: "mysql.example.com:3306"
        connectProtocol: "tcp"
        databaseName: "cadence"
        tls:
          enabled: true
          sslmode: "require"  # Options: disable, require, verify-ca, verify-full
          caFile: "/etc/cadence/tls/mysql-ca.pem"
          certFile: "/etc/cadence/tls/mysql-client-cert.pem"
          keyFile: "/etc/cadence/tls/mysql-client-key.pem"

PostgreSQL

persistence:
  datastores:
    default:
      sql:
        pluginName: "postgres"
        connectAddr: "postgres.example.com:5432"
        connectProtocol: "tcp"
        databaseName: "cadence"
        tls:
          enabled: true
          sslmode: "verify-full"  # Options: disable, require, verify-ca, verify-full
          caFile: "/etc/cadence/tls/postgres-ca.pem"

Use verify-full for PostgreSQL and verify-ca for MySQL in production to prevent MITM attacks.

Kafka Security (SASL)

For Kafka replication and async workflows:

SASL/PLAIN

kafka:
  clusters:
    test:
      brokers:
        - "kafka1:9093"
        - "kafka2:9093"
      tls:
        enabled: true
        caFile: "/etc/cadence/tls/kafka-ca.crt"
      sasl:
        enabled: true
        mechanism: "PLAIN"
        user: "cadence-user"
        password: "${KAFKA_PASSWORD}"

SASL/SCRAM

kafka:
  clusters:
    test:
      brokers:
        - "kafka1:9093"
      sasl:
        enabled: true
        mechanism: "SCRAM-SHA-512"  # or "SCRAM-SHA-256"
        user: "cadence-user"
        password: "${KAFKA_PASSWORD}"

For SASL support, see common/authorization/scram_client.go for implementation details.

Network Security

Port Configuration

Default ports:

Service	TChannel	gRPC	HTTP	Metrics
Frontend	7933	7833	-	9090
History	7934	-	-	9091
Matching	7935	-	-	9092
Worker	7939	-	-	9093

Firewall Rules

Inbound Rules

# Frontend (from clients)
Allow TCP 7933 from 0.0.0.0/0  # TChannel
Allow TCP 7833 from 0.0.0.0/0  # gRPC

# Inter-service (within cluster)
Allow TCP 7933-7939 from <cluster_cidr>

# Metrics (from Prometheus)
Allow TCP 9090-9093 from <monitoring_cidr>

Outbound Rules

# Database
Allow TCP 9042 to <cassandra_hosts>   # Cassandra
Allow TCP 3306 to <mysql_hosts>        # MySQL
Allow TCP 5432 to <postgres_hosts>     # PostgreSQL

# Kafka
Allow TCP 9092-9093 to <kafka_brokers>

# Cross-DC replication
Allow TCP 7833 to <remote_cluster_cidr>

Network Isolation

Best practices:

Deploy in private subnets - No direct internet access
Use NAT gateway - For outbound internet (archival to S3/GCS)
VPC peering - For cross-region/cross-account replication
Security groups - Restrict access by source IP/security group

Secrets Management

Environment Variables

Use environment variable substitution for secrets:

persistence:
  datastores:
    default:
      cassandra:
        user: "${CASSANDRA_USER}"
        password: "${CASSANDRA_PASSWORD}"

kafka:
  clusters:
    test:
      sasl:
        user: "${KAFKA_USER}"
        password: "${KAFKA_PASSWORD}"

Secrets Management Systems

Integrate with external secret stores:

AWS Secrets Manager

# Fetch secrets at startup
export MYSQL_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id cadence/mysql/password \
  --query SecretString \
  --output text)

./cadence-server start

HashiCorp Vault

# Fetch secrets from Vault
export CASSANDRA_PASSWORD=$(vault kv get \
  -field=password \
  secret/cadence/cassandra)

./cadence-server start

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: cadence-secrets
type: Opaque
data:
  mysql-password: <base64-encoded>
  kafka-password: <base64-encoded>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cadence-frontend
spec:
  template:
    spec:
      containers:
      - name: frontend
        env:
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cadence-secrets
              key: mysql-password

Audit Logging

Enable Audit Logs

log:
  level: "info"
  encoding: "json"
  outputFile: "/var/log/cadence/audit.log"

Audit log entries include:

{
  "ts": "2026-03-04T10:15:30.123Z",
  "level": "info",
  "msg": "API request",
  "operation": "StartWorkflowExecution",
  "domain": "my-domain",
  "principal": "[email protected]",
  "remote-addr": "10.0.1.42",
  "wf-id": "workflow-123"
}

Audit Events

Key events to monitor:

Workflow start/terminate/cancel
Domain registration/update
Authorization failures
Admin operations (shard management, queue operations)

Ship audit logs to a centralized logging system (Splunk, Elasticsearch) for compliance and forensics.

Compliance

Data Encryption

At Rest

Database: Enable encryption at rest (Cassandra transparent encryption, RDS encryption)
Archival: Enable S3 SSE or GCS encryption
Local storage: Use encrypted volumes (LUKS, EBS encryption)

In Transit

Client ↔ Frontend: TLS/mTLS
Inter-service: TLS (optional but recommended)
Database connections: TLS
Kafka: TLS + SASL

Data Retention

domainDefaults:
  archival:
    history:
      status: "enabled"
      URI: "s3://my-bucket/cadence-archival"
    visibility:
      status: "enabled"
      URI: "s3://my-bucket/cadence-visibility"
  retention: "7d"  # Primary storage retention

Workflow history is retained in primary storage for the retention period, then archived. Archived data is retained indefinitely unless you implement archival bucket lifecycle policies.

Security Best Practices

1. Principle of Least Privilege

Use separate database users per service
Grant minimal required permissions
Rotate credentials regularly

2. Network Segmentation

Deploy services in private subnets
Use security groups/network policies
Isolate production from non-production

3. TLS Everywhere

Enable TLS for all RPC communication
Use mTLS for inter-cluster replication
Enable TLS for database connections

4. Secrets Rotation

Rotate JWT signing keys annually
Rotate database passwords quarterly
Rotate TLS certificates before expiry

5. Monitoring and Alerting

# Failed authorization attempts
rate(cadence_authorization_failed[5m]) > 10

# TLS handshake failures
rate(cadence_tls_handshake_errors[5m]) > 5

Troubleshooting

Authorization Failures

Check JWT validation:

# Decode JWT
echo $JWT_TOKEN | cut -d. -f2 | base64 -d | jq

# Verify expiration
date -d @$(echo $JWT_TOKEN | cut -d. -f2 | base64 -d | jq -r .exp)

TLS Connection Issues

Verify certificate chain:

# Test TLS connection
openssl s_client -connect cadence-frontend:7833 \
  -CAfile /etc/cadence/tls/ca.crt \
  -cert /etc/cadence/tls/client.crt \
  -key /etc/cadence/tls/client.key

# Verify certificate
openssl x509 -in /etc/cadence/tls/server.crt -text -noout

Database TLS Issues

Test database connectivity:

# MySQL
mysql -h mysql.example.com \
  --ssl-ca=/etc/cadence/tls/mysql-ca.pem \
  --ssl-cert=/etc/cadence/tls/mysql-client-cert.pem \
  --ssl-key=/etc/cadence/tls/mysql-client-key.pem

# PostgreSQL
psql "host=postgres.example.com \
  sslmode=verify-full \
  sslrootcert=/etc/cadence/tls/postgres-ca.pem \
  dbname=cadence"

Get Started

Core Concepts

Architecture

Deployment

Operations

Client SDKs

​Overview

​Authorization

​OAuth Authorization

​Configuration

​JWT Public Key Configuration

​Using External Identity Providers

​JWT Claims

​Authorization Validation

​Client Authentication

​TLS/SSL Configuration

​Server-Side TLS

​TLS Configuration Options

​Mutual TLS (mTLS)

​Server Configuration

​Client Configuration

​Cross-Cluster TLS

​Database Security

​Cassandra TLS

​MySQL/PostgreSQL TLS

​MySQL

​PostgreSQL

​Kafka Security (SASL)

​SASL/PLAIN

​SASL/SCRAM

​Network Security

​Port Configuration

​Firewall Rules

​Inbound Rules

​Outbound Rules

​Network Isolation

​Secrets Management

​Environment Variables

​Secrets Management Systems

​AWS Secrets Manager

​HashiCorp Vault

​Kubernetes Secrets

​Audit Logging

​Enable Audit Logs

​Audit Events

​Compliance

​Data Encryption

​At Rest

​In Transit

​Data Retention

​Security Best Practices

​1. Principle of Least Privilege

​2. Network Segmentation

​3. TLS Everywhere

​4. Secrets Rotation

​5. Monitoring and Alerting

​Troubleshooting

​Authorization Failures

​TLS Connection Issues

​Database TLS Issues

​See Also

Build docs developers (and LLMs) love

Overview

Authorization

OAuth Authorization

Configuration

JWT Public Key Configuration

Using External Identity Providers

JWT Claims

Authorization Validation

Client Authentication

TLS/SSL Configuration

Server-Side TLS

TLS Configuration Options

Mutual TLS (mTLS)

Server Configuration

Client Configuration

Cross-Cluster TLS

Database Security

Cassandra TLS

MySQL/PostgreSQL TLS

MySQL

PostgreSQL

Kafka Security (SASL)

SASL/PLAIN

SASL/SCRAM

Network Security

Port Configuration

Firewall Rules

Inbound Rules

Outbound Rules

Network Isolation

Secrets Management

Environment Variables

Secrets Management Systems

AWS Secrets Manager

HashiCorp Vault

Kubernetes Secrets

Audit Logging

Enable Audit Logs

Audit Events

Compliance

Data Encryption

At Rest

In Transit

Data Retention

Security Best Practices

1. Principle of Least Privilege

2. Network Segmentation

3. TLS Everywhere

4. Secrets Rotation

5. Monitoring and Alerting

Troubleshooting

Authorization Failures

TLS Connection Issues

Database TLS Issues

See Also