Monitoring and Observability

Overview

Cadence provides extensive monitoring capabilities through metrics, structured logging, and health checks. This guide covers production monitoring strategies, metric integration, and observability best practices.

Metrics Architecture

Cadence supports multiple metrics backends through the Tally metrics library:

Supported Reporters

Prometheus - Recommended for production use
StatsD - Legacy support with tag limitations
M3 - Uber’s internal metrics platform

Only one metrics reporter can be configured per service. Attempting to configure multiple reporters will result in a fatal error.

Prometheus Integration

Configuration

Configure Prometheus metrics in your config.yaml:

services:
  frontend:
    metrics:
      prometheus:
        timerType: "histogram"
        listenAddress: "0.0.0.0:9090"
        defaultHistogramBuckets:
          - 0.001
          - 0.005
          - 0.01
          - 0.05
          - 0.1
          - 0.5
          - 1.0
          - 5.0
          - 10.0
      tags:
        environment: "production"
        datacenter: "us-east-1"
      prefix: "cadence"
      reportingInterval: "1s"

Configuration Parameters

Parameter	Description	Default
`listenAddress`	Host:Port for Prometheus scrape endpoint	Required
`timerType`	Use “histogram” for latency metrics	”histogram”
`defaultHistogramBuckets`	Histogram bucket boundaries in seconds	See above
`reportingInterval`	Metric reporting interval	1s
`tags`	Global tags applied to all metrics
`prefix`	Metric name prefix	""

Metric Sanitization

Cadence automatically sanitizes metric names to comply with Prometheus naming conventions:

Characters - and . are replaced with _
Only alphanumeric characters and _ are allowed
Tag names follow the same rules

Metric names emitted may differ from internal metric names due to sanitization. Ensure your dashboards account for this transformation.

StatsD Integration

Configuration

services:
  history:
    metrics:
      statsd:
        hostPort: "127.0.0.1:8125"
        prefix: "cadence.history"
        flushInterval: "10s"
        flushBytes: 512
      tags:
        service: "history"

Tally’s standard StatsD implementation doesn’t support tagging. Cadence provides an enhanced reporter with tag support.

Key Metrics

Service Health Metrics

These metrics are emitted by all Cadence services:

# Process metrics
cadence_restarts                    # Service restart counter
cadence_num_goroutines             # Active goroutines
cadence_gomaxprocs                 # GOMAXPROCS setting

# Memory metrics
cadence_memory_allocated           # Total allocated bytes
cadence_memory_heap                # Heap memory
cadence_memory_heapidle            # Idle heap memory
cadence_memory_heapinuse           # In-use heap memory
cadence_memory_stack               # Stack memory
cadence_memory_num_gc              # GC run count
cadence_memory_gc_pause_ms         # GC pause duration

Persistence Metrics

Monitor database operations across all persistence calls:

# Operation metrics (per operation type)
cadence_persistence_requests
cadence_persistence_errors
cadence_persistence_latency

# Example operations:
# - PersistenceCreateWorkflowExecution
# - PersistenceUpdateWorkflowExecution
# - PersistenceGetWorkflowExecution
# - PersistenceAppendHistoryEvents

RPC Metrics

Track inter-service and client communication:

# Client metrics (Frontend, History, Matching)
cadence_frontend_client_requests
cadence_frontend_client_errors
cadence_frontend_client_latency

cadence_history_client_requests
cadence_history_client_errors
cadence_history_client_latency

cadence_matching_client_requests
cadence_matching_client_errors
cadence_matching_client_latency

Workflow Execution Metrics

# Workflow lifecycle
cadence_workflow_started
cadence_workflow_completed
cadence_workflow_failed
cadence_workflow_timeout
cadence_workflow_canceled
cadence_workflow_continued_as_new

# Task processing
cadence_decision_task_schedule_to_start_latency
cadence_activity_task_schedule_to_start_latency
cadence_decision_task_execution_latency
cadence_activity_task_execution_latency

Service-Specific Metrics

History Service

cadence_history_cache_requests
cadence_history_cache_errors
cadence_history_cache_latency
cadence_history_shard_context_closed
cadence_history_replication_tasks_applied

Matching Service

cadence_matching_tasks_added
cadence_matching_tasks_dispatched
cadence_matching_poll_success
cadence_matching_poll_timeout
cadence_matching_tasklist_backlog

Metric Tags

Cadence automatically applies these standard tags:

Tag	Description	Example
`cadence_service`	Service name	”frontend”, “history”
`operation`	API operation	”StartWorkflowExecution”
`domain`	Workflow domain	”my-domain”
`tasklist`	Task list name	”my-tasklist”
`workflow_type`	Workflow type	”MyWorkflow”
`activity_type`	Activity type	”MyActivity”

Health Checks

Endpoint Configuration

Cadence services expose health check endpoints:

GET http://<host>:<port>/health

Response Format

{
  "ok": true,
  "msg": "All systems operational"
}

Health Check Implementation

Health checks verify:

Service startup - Service has initialized successfully
Persistence connectivity - Database connections are healthy
Ring membership - Service is part of the ring

Integrate health checks with your load balancer and orchestration platform (Kubernetes, ECS) for automated failover.

Logging Configuration

Structured Logging

Cadence uses Zap for structured JSON logging:

log:
  level: "info"
  encoding: "json"
  outputFile: "/var/log/cadence/cadence.log"
  levelKey: "level"

Log Levels

Level	Use Case
`debug`	Development and troubleshooting
`info`	Normal operations (recommended for production)
`warn`	Warning conditions
`error`	Error conditions
`fatal`	Fatal errors causing service shutdown

Console vs JSON Encoding

# JSON format (production)
log:
  encoding: "json"
  outputFile: "/var/log/cadence/frontend.log"

# Console format (development)
log:
  encoding: "console"
  stdout: true

Log Fields

Cadence includes contextual fields in all log entries:

{
  "ts": "2026-03-04T10:15:30.123Z",
  "level": "info",
  "msg": "Workflow execution started",
  "logger": "history.engine",
  "wf-domain-name": "my-domain",
  "wf-id": "workflow-123",
  "wf-run-id": "abc-def-ghi",
  "shard-id": 42
}

Alerting Guidelines

Critical Alerts

Configure alerts for these conditions:

Service Availability

# Service down
up{job="cadence-frontend"} == 0

# High restart rate
rate(cadence_restarts[5m]) > 0.1

Persistence Issues

# High error rate
rate(cadence_persistence_errors[5m]) > 10

# High latency
histogram_quantile(0.99, cadence_persistence_latency) > 1.0

Task Processing

# Task list backlog growth
rate(cadence_matching_tasklist_backlog[5m]) > 100

# High task timeout rate
rate(cadence_workflow_timeout[5m]) > 5

Warning Alerts

# Memory usage
cadence_memory_heap > 8e9  # 8GB

# High GC pause time
rate(cadence_memory_gc_pause_ms[5m]) > 100

# High request latency
histogram_quantile(0.95, cadence_frontend_client_latency) > 0.5

Distributed Tracing

Cadence supports distributed tracing through YARPC:

Configuration

services:
  frontend:
    rpc:
      grpcPort: 7833
      # Tracing is enabled via YARPC configuration

Distributed tracing integration requires custom YARPC middleware configuration. Refer to YARPC documentation for Jaeger or Zipkin integration.

Monitoring Best Practices

Dashboard Organization

Create dashboards for each layer:

Service Health - CPU, memory, goroutines, restarts
RPC Layer - Request rates, errors, latency by service and operation
Persistence Layer - Database operations, latency, errors
Workflow Execution - Started/completed/failed workflows, task latency
Business Metrics - Domain-specific workflow metrics

Cardinality Management

Avoid high-cardinality dimensions in metric tags:

Workflow IDs
Run IDs
Specific timestamps
User IDs

Use workflow_type, domain, and operation instead.

Retention Policies

Raw metrics: 15-30 days
Aggregated metrics: 90-365 days
Logs: 7-30 days (ship to archival for longer retention)

Troubleshooting

No Metrics Appearing

Verify metrics endpoint is accessible:
```
curl http://localhost:9090/metrics
```

Check service logs for metrics initialization:

grep "metric" /var/log/cadence/frontend.log

Validate configuration:

# Only one reporter should be configured
metrics:
  prometheus:  # OR statsd OR m3, not multiple
    listenAddress: "0.0.0.0:9090"

High Metric Cardinality

If Prometheus complains about high cardinality:

Review custom tags configuration
Ensure workflow IDs aren’t being used as tag values
Check for runaway domain/tasklist creation

Missing Log Output

Check log configuration:

log:
  level: "info"  # Not "INFO"
  outputFile: "/var/log/cadence/service.log"  # Ensure path exists and is writable

Get Started

Core Concepts

Architecture

Deployment

Operations

Client SDKs

​Overview

​Metrics Architecture

​Supported Reporters

​Prometheus Integration

​Configuration

​Configuration Parameters

​Metric Sanitization

​StatsD Integration

​Configuration

​Key Metrics

​Service Health Metrics

​Persistence Metrics

​RPC Metrics

​Workflow Execution Metrics

​Service-Specific Metrics

​History Service

​Matching Service

​Metric Tags

​Health Checks

​Endpoint Configuration

​Response Format

​Health Check Implementation

​Logging Configuration

​Structured Logging

​Log Levels

​Console vs JSON Encoding

​Log Fields

​Alerting Guidelines

​Critical Alerts

​Service Availability

​Persistence Issues

​Task Processing

​Warning Alerts

​Distributed Tracing

​Configuration

​Monitoring Best Practices

​Dashboard Organization

​Cardinality Management

​Retention Policies

​Troubleshooting

​No Metrics Appearing

​High Metric Cardinality

​Missing Log Output

​See Also

Build docs developers (and LLMs) love

Overview

Metrics Architecture

Supported Reporters

Prometheus Integration

Configuration

Configuration Parameters

Metric Sanitization

StatsD Integration

Configuration

Key Metrics

Service Health Metrics

Persistence Metrics

RPC Metrics

Workflow Execution Metrics

Service-Specific Metrics

History Service

Matching Service

Metric Tags

Health Checks

Endpoint Configuration

Response Format

Health Check Implementation

Logging Configuration

Structured Logging

Log Levels

Console vs JSON Encoding

Log Fields

Alerting Guidelines

Critical Alerts

Service Availability

Persistence Issues

Task Processing

Warning Alerts

Distributed Tracing

Configuration

Monitoring Best Practices

Dashboard Organization

Cardinality Management

Retention Policies

Troubleshooting

No Metrics Appearing

High Metric Cardinality

Missing Log Output

See Also