Skip to main content

​
Overview

Cadence is a distributed system consisting of multiple service components that work together to provide durable workflow execution. This guide covers server architecture, prerequisites, and deployment steps for production environments.

​
Architecture

Cadence server consists of four core services:
  • Frontend: Handles all client-facing API requests and rate limiting
  • History: Maintains workflow execution state and event histories
  • Matching: Routes tasks between workflows and workers via task lists
  • Worker: Performs system workflows like archival and cross-datacenter replication
Additionally, an optional Shard Distributor service manages shard distribution in larger deployments.

​
Prerequisites

​
System Requirements

  • Operating System: Linux (recommended), macOS (development)
  • Go: Version 1.23+ (if building from source)
  • Database: One of the following:
    • Cassandra 4.x or ScyllaDB
    • MySQL 8.0+
    • PostgreSQL 12+
  • Memory: Minimum 4GB RAM per service instance
  • CPU: 2+ cores per service instance

​
Network Ports

Default ports used by Cadence services:
ServiceTChannel PortgRPC PortMetrics Portpprof Port
Frontend7933783380007936
History7934783480017937
Matching7935783580027938
Worker7939-80037940

​
External Dependencies

  • Persistence: Database for storing workflow state (required)
  • Ringpop: Built-in cluster membership and service discovery
  • Metrics: Prometheus or StatsD (recommended)
  • Visibility: ElasticSearch/OpenSearch for advanced workflow search (optional)
  • Archival: S3, GCS, or file storage for workflow history archival (optional)

​
Installation

​
Using Pre-built Binaries

Install Cadence using Homebrew (macOS/Linux): 
brew install cadence-workflow
This installs:
  • cadence-server - Main server binary
  • cadence - CLI tool
  • cadence-sql-tool - SQL schema management
  • cadence-cassandra-tool - Cassandra schema management

​
Building from Source

git clone https://github.com/cadence-workflow/cadence.git
cd cadence
make cadence-server
The binary will be available at .build/bin/cadence-server.

​
Server Deployment

1

Set up the database

Initialize your persistence layer. See Database Setup for detailed instructions.
# For Cassandra
make install-schema

# For MySQL
make install-schema-mysql

# For PostgreSQL
make install-schema-postgres
2

Create configuration file

Create a YAML configuration file based on your environment. See Configuration for all options.
cp config/development.yaml config/production.yaml
# Edit config/production.yaml with your settings
3

Start services

Launch Cadence services. You can run all services in a single process or separate processes.Single Process (All Services):
cadence-server --root config --env production start \
  --services=frontend,matching,history,worker
Separate Processes:
# Frontend
cadence-server --root config --env production start --services=frontend

# History
cadence-server --root config --env production start --services=history

# Matching
cadence-server --root config --env production start --services=matching

# Worker
cadence-server --root config --env production start --services=worker
4

Verify deployment

Check that services are running and healthy:
# Using CLI
cadence --address <frontend-host>:7933 cluster health

# Check service metrics
curl http://<frontend-host>:8000/metrics
5

Register a domain

Create your first domain to start using Cadence:
cadence --domain my-domain domain register \
  --retention 7 \
  --description "Production workflows"

​
Service Discovery and Clustering

Cadence uses Ringpop for cluster membership and service discovery:
config/production.yaml
ringpop:
  name: cadence
  bootstrapMode: hosts
  bootstrapHosts:
    - "10.0.1.10:7933"  # Frontend
    - "10.0.1.11:7934"  # History
    - "10.0.1.12:7935"  # Matching
  maxJoinDuration: 30s
All service instances must be able to communicate with each other on their TChannel ports for cluster membership to work properly.

​
Production Considerations

​
High Availability

  • Run multiple instances of each service type (minimum 3 for production)
  • Use a load balancer in front of Frontend services for client requests
  • Deploy across availability zones for fault tolerance
  • History shards are distributed across History service instances automatically

​
Monitoring

Enable metrics collection for observability:
config/production.yaml
services:
  frontend:
    metrics:
      prometheus:
        timerType: "histogram"
        listenAddress: "0.0.0.0:8000"
  # Repeat for other services...
Key metrics to monitor:
  • Request latency and error rates
  • Workflow execution counts
  • Task queue depth
  • Persistence latency
  • System resource usage (CPU, memory, disk I/O)

​
Resource Tuning

History Shards: Configure based on expected scale: 
persistence:
  numHistoryShards: 1024  # Production: 1024-4096
The number of history shards cannot be changed after cluster initialization. Choose carefully based on your expected maximum cluster size.
Connection Pools: Tune database connection pools: 
datastores:
  default:
    sql:
      maxConns: 20          # Max connections per host
      maxIdleConns: 20      # Idle connection pool
      maxConnLifetime: "1h" # Connection lifetime

​
Security

  • TLS/mTLS: Enable encrypted communication between services and clients
  • Authentication: Configure OAuth or JWT-based authentication
  • Network segmentation: Restrict database access to Cadence services only
  • Secret management: Use environment variables or secret managers for credentials

​
Logging

Configure appropriate log levels: 
log:
  stdout: true
  level: "info"  # Production: info or warn
  levelKey: "level"
For production, avoid debug level logging to reduce overhead.

​
Backup and Recovery

  • Database backups: Regular automated backups of persistence layer
  • Point-in-time recovery: Enable database PITR capabilities
  • Disaster recovery: Document and test recovery procedures
  • Configuration management: Version control all configuration files

​
Upgrading Cadence

1

Review release notes

Check the CHANGELOG for breaking changes and migration requirements.
2

Update database schema

Apply schema updates before deploying new server version:
# Check current schema version
cadence-sql-tool --ep <db-host> --db cadence show-version

# Apply updates
cadence-sql-tool --ep <db-host> --db cadence update-schema \
  --version <target-version>
3

Perform rolling upgrade

Upgrade services one at a time to maintain availability:
  1. Update Worker services first
  2. Update Matching services
  3. Update History services
  4. Update Frontend services last
4

Verify after upgrade

Confirm all services are healthy and processing requests normally.

​
Troubleshooting

​
Services Won’t Start

  1. Check database connectivity
  2. Verify configuration file syntax
  3. Ensure all required ports are available
  4. Check logs for specific error messages

​
Cluster Membership Issues

  1. Verify network connectivity between all service instances
  2. Check Ringpop bootstrap hosts configuration
  3. Ensure TChannel ports are not blocked by firewalls

​
Performance Issues

  1. Check database performance and connection pool utilization
  2. Monitor service metrics for bottlenecks
  3. Review history shard distribution across instances
  4. Consider scaling out services horizontally

​
Next Steps

Configuration

Deep dive into configuration options

Database Setup

Detailed database configuration

Docker Deployment

Deploy using Docker containers

Kubernetes

Production Kubernetes deployment

Build docs developers (and LLMs) love

Get started for freeTalk to us