Batch Commands - Cadence

The batch command group provides operations for performing actions on multiple workflows simultaneously based on queries.

Command Overview

cadence workflow batch [command] [flags]

Batch operations allow you to:

Terminate multiple workflows
Signal multiple workflows
Cancel multiple workflows
Monitor batch job progress

Batch operations run as workflows in the special cadence-batcher domain. This provides fault tolerance and progress tracking.

Starting Batch Jobs

batch start

Start a new batch operation job.

cadence workflow batch start \
  --query "WorkflowType='MyWorkflow' AND CloseStatus='Failed'" \
  --reason "Recovering from deployment issue" \
  --batch_type terminate

--query

string

required

SQL-like query to select target workflows

--reason

string

required

Reason for the batch operation (for auditing)

--batch_type

string

required

Operation type: terminate, cancel, or signal

--rps

int

default:"50"

Rate limit in operations per second

--pagesize

int

default:"1000"

Page size for processing workflows

--concurrency

int

default:"5"

Number of concurrent operations

--yes

bool

Skip confirmation prompt

--retry_attempts

int

default:"3"

Retry attempts for retriable errors

--heart_beat_timeout_seconds

int

default:"600"

Activity heartbeat timeout in seconds

--max_activity_retries

int

default:"0"

Max activity retries before workflow retry (0 = unlimited)

Batch Types

Terminate Batch

Terminate multiple workflows.

cadence workflow batch start \
  --query "CloseStatus='Running' AND StartTime < '2024-01-01'" \
  --batch_type terminate \
  --reason "Cleaning up old workflows"

Cancel Batch

Gracefully cancel multiple workflows.

cadence workflow batch start \
  --query "WorkflowType='OrderProcessing' AND CustomStatus='Pending'" \
  --batch_type cancel \
  --reason "Order batch cancelled by user"

Signal Batch

Send signals to multiple workflows.

cadence workflow batch start \
  --query "WorkflowType='UserWorkflow' AND IsActive=true" \
  --batch_type signal \
  --signal_name UpdateConfig \
  --input '{"feature_flag": true}' \
  --reason "Enabling new feature"

--signal_name

string

Signal name (required for signal batch type)

--input

string

Signal payload as JSON (optional)

Monitoring Batch Jobs

batch describe

Check batch job status and progress.

cadence workflow batch describe --job_id job-uuid

Example Output (Running):

{
  "msg": "batch job is running",
  "progress": {
    "totalEstimate": 1000,
    "currentPage": 5,
    "succeedCount": 473,
    "failedCount": 2,
    "pageSize": 100,
    "ratePerSecond": 50
  }
}

Example Output (Completed):

{
  "msg": "batch job is finished successfully"
}

--job_id

string

required

Batch job ID returned from start command

batch list

List batch jobs for a domain.

cadence workflow batch list --pagesize 30

Example Output:

[
  {
    "jobID": "job-uuid-1",
    "startTime": "2024-01-15T10:00:00Z",
    "reason": "Recovering from deployment",
    "operator": "[email protected]",
    "status": "COMPLETED",
    "closeTime": "2024-01-15T10:15:00Z"
  },
  {
    "jobID": "job-uuid-2",
    "startTime": "2024-01-15T11:00:00Z",
    "reason": "Batch signal update",
    "operator": "[email protected]",
    "status": "RUNNING"
  }
]

--pagesize

int

default:"30"

Number of results to return

Stopping Batch Jobs

batch terminate

Stop a running batch job.

cadence workflow batch terminate \
  --job_id job-uuid \
  --reason "Operation no longer needed"

--job_id

string

required

Batch job ID to terminate

--reason

string

required

Reason for termination

Query Examples

By Workflow Type

cadence workflow batch start \
  --query "WorkflowType='ProcessOrder'" \
  --batch_type terminate \
  --reason "Maintenance"

By Time Range

cadence workflow batch start \
  --query "StartTime BETWEEN '2024-01-01T00:00:00Z' AND '2024-01-31T23:59:59Z'" \
  --batch_type signal \
  --signal_name Archive \
  --reason "Monthly archival"

By Status

cadence workflow batch start \
  --query "CloseStatus='Failed' AND WorkflowType='DataSync'" \
  --batch_type terminate \
  --reason "Cleanup failed syncs"

By Custom Search Attributes

cadence workflow batch start \
  --query "CustomerId='12345' AND Status='Pending'" \
  --batch_type signal \
  --signal_name ProcessNow \
  --reason "Priority processing"

Complex Queries

cadence workflow batch start \
  --query "WorkflowType='OrderWorkflow' AND (CloseStatus='Failed' OR CustomStatus='Stalled') AND StartTime > '2024-01-01'" \
  --batch_type terminate \
  --reason "Cleanup problematic orders"

Rate Limiting and Performance

Conservative Settings

For production systems under load:

cadence workflow batch start \
  --query "..." \
  --batch_type terminate \
  --rps 10 \
  --concurrency 2 \
  --reason "..."

Aggressive Settings

For dedicated maintenance windows:

cadence workflow batch start \
  --query "..." \
  --batch_type signal \
  --rps 500 \
  --concurrency 20 \
  --pagesize 2000 \
  --reason "..."

Monitoring Progress

Continuously monitor a batch job:

# Get job ID from start command
JOB_ID="job-uuid"

# Monitor progress
while true; do
  cadence workflow batch describe --job_id $JOB_ID
  sleep 10
done

Error Handling

Viewing Failed Operations

The batch job tracks failures in its progress:

cadence workflow batch describe --job_id job-uuid

Look for failedCount in the output.

Retry Configuration

cadence workflow batch start \
  --query "..." \
  --batch_type signal \
  --retry_attempts 5 \
  --max_activity_retries 3 \
  --reason "..."

retry_attempts: Retries for individual workflow operations
max_activity_retries: Activity retries before workflow retry

Common Use Cases

Clean Up Old Workflows

cadence workflow batch start \
  --query "CloseTime < '2023-01-01' AND CloseStatus='Completed'" \
  --batch_type terminate \
  --reason "Archival cleanup" \
  --rps 100

Emergency Stop

cadence workflow batch start \
  --query "WorkflowType='ProblematicWorkflow'" \
  --batch_type terminate \
  --reason "Emergency stop due to bug" \
  --yes

Feature Flag Update

cadence workflow batch start \
  --query "WorkflowType='FeatureEnabledWorkflow' AND CloseStatus='Running'" \
  --batch_type signal \
  --signal_name UpdateFeatureFlag \
  --input '{"feature_x": true}' \
  --reason "Enabling feature X rollout"

Graceful Shutdown

cadence workflow batch start \
  --query "CustomStatus='Active' AND Environment='staging'" \
  --batch_type cancel \
  --reason "Staging environment shutdown" \
  --rps 50

Best Practices

Test Queries: Use cadence workflow count to verify query before batch operation
Dry Run: Start with small rps and concurrency values
Monitor: Watch progress with batch describe regularly
Off-Peak: Schedule large batch operations during low-traffic periods
Documentation: Always provide descriptive reason for auditing
Confirmation: Use --yes flag only in automated scripts

Verification

Before Starting Batch

Verify the query selects the correct workflows:

# Count matching workflows
cadence workflow count --query "WorkflowType='MyWorkflow' AND CloseStatus='Failed'"

# List sample workflows
cadence workflow list --query "WorkflowType='MyWorkflow' AND CloseStatus='Failed'" --pagesize 5

During Batch Operation

# Check progress
cadence workflow batch describe --job_id job-uuid

# Verify target workflows
cadence workflow list --query "<your-query>" --pagesize 10

After Batch Operation

# Confirm completion
cadence workflow batch describe --job_id job-uuid

# Verify results
cadence workflow count --query "<your-query>"

Troubleshooting

Batch job stuck

Problem: Progress not advancingSolutions:

Check batch job workflow: cadence workflow describe -w job-uuid --domain cadence-batcher
Verify workers are running for cadence-batcher domain
Check server logs for errors

High failure rate

Problem: Many operations failingCauses:

Workflows already terminated/completed
Invalid signal names
Rate limiting on server

Solutions:

Review query to ensure correct workflows
Verify signal handler exists in workflow code
Reduce rps and concurrency

Batch job not starting

Problem: Start command succeeds but no progressSolutions:

Verify cadence-batcher domain exists
Check if workers are polling cadence-batcher-tasklist
Review server logs for startup errors

Batch Job Lifecycle

Started: Job workflow created, query validation begins
Running: Processing workflows page by page
Completed: All workflows processed successfully
Failed: Unrecoverable error occurred
Terminated: Manually stopped via terminate command

Next Steps

Workflow Commands

Individual workflow operations

Workflow Reset

Batch reset operations

Admin Commands

Administrative operations

Operations Guide

Operations best practices

Overview

Commands

​Command Overview

​Starting Batch Jobs

​batch start

​Batch Types

​Terminate Batch

​Cancel Batch

​Signal Batch

​Monitoring Batch Jobs

​batch describe

​batch list

​Stopping Batch Jobs

​batch terminate

​Query Examples

​By Workflow Type

​By Time Range

​By Status

​By Custom Search Attributes

​Complex Queries

​Rate Limiting and Performance

​Conservative Settings

​Aggressive Settings

​Monitoring Progress

​Error Handling

​Viewing Failed Operations

​Retry Configuration

​Common Use Cases

​Clean Up Old Workflows

​Emergency Stop

​Feature Flag Update

​Graceful Shutdown

​Best Practices

​Verification

​Before Starting Batch

​During Batch Operation

​After Batch Operation

​Troubleshooting

​Batch Job Lifecycle

​Next Steps

Workflow Commands

Workflow Reset

Admin Commands

Operations Guide

Build docs developers (and LLMs) love

Command Overview

Starting Batch Jobs

batch start

Batch Types

Terminate Batch

Cancel Batch

Signal Batch

Monitoring Batch Jobs

batch describe

batch list

Stopping Batch Jobs

batch terminate

Query Examples

By Workflow Type

By Time Range

By Status

By Custom Search Attributes

Complex Queries

Rate Limiting and Performance

Conservative Settings

Aggressive Settings

Monitoring Progress

Error Handling

Viewing Failed Operations

Retry Configuration

Common Use Cases

Clean Up Old Workflows

Emergency Stop

Feature Flag Update

Graceful Shutdown

Best Practices

Verification

Before Starting Batch

During Batch Operation

After Batch Operation

Troubleshooting

Batch Job Lifecycle

Next Steps