Chasm Architecture

Chasm (Coordinated Heterogeneous Application State Machines) is Temporal’s library for building complex, durable, distributed applications using a hierarchical state machine model. It provides a framework for implementing Temporal features that require sophisticated state management beyond traditional workflows.

Chasm is an internal implementation framework, not a user-facing API. It is used to implement features like Scheduler, Worker Deployment, and other advanced Temporal capabilities.

Motivation

Temporal initially built all features using the core Workflow concept. However, certain features required:

Complex state machines with multiple concurrent components
Long-lived executions with evolving requirements
Hierarchical composition of state machines
Sophisticated lifecycle management
Rich interaction patterns beyond workflow signals/queries

Chasm was created to provide a structured framework for building these features while maintaining Temporal’s durability and consistency guarantees.

Core Concepts

Terminology

Chasm introduces several key concepts:

Archetype

An Archetype is a type of state machine (e.g., Workflow, Activity, Scheduler).Each archetype defines:

State structure
Allowed transitions
Task types
Request-reply handlers

Execution

An Execution is an instance of an archetype (e.g., a specific workflow execution, a specific scheduler).Each execution has:

A BusinessID (user-meaningful identifier)
Unique across non-terminal executions in a namespace
Can consist of multiple runs

Component

A Component is a node in the execution tree.An execution is structured as a tree where:

Root component represents the execution
Child components represent sub-state machines
Each component has its own state and tasks

ComponentRef

A ComponentRef uniquely identifies a component within an execution.Contains:

ExecutionKey (namespace + business ID)
ComponentPath (tree path to component)
Transition information (for multi-cluster failover)

Architecture

Hierarchical State Machines

Chasm applications are structured as trees of components:

Root component: Represents the overall execution
Child components: Represent sub-state machines
Leaf components: Have no children

Each component:

Has its own state (persisted)
Schedules its own tasks (timers, side effects)
Handles requests (via request-reply pattern)
Can create/delete child components dynamically

State Management

Component state is persisted using Temporal’s existing infrastructure:

Stored in Mutable State (same as workflow state)
Uses event sourcing (state changes generate events)
Atomic updates via database transactions
Cached for performance

Task Processing

Components schedule tasks for execution: Timer Tasks:

Execute at a specific time
Drive time-based state transitions
Handled by History Service timer queue

Side Effect Tasks:

Make external service calls
Execute immediately but with retries
Handled by History Service transfer queue

Component Lifecycle

Component Creation

Components are created dynamically:

// Pseudo-code
func (s *Scheduler) OnTriggerRequest(req TriggerRequest) {
    // Create a new Backfiller component
    backfiller := s.CreateChildComponent(
        "Backfiller",
        BackfillerState{
            BackfillID: generateID(),
            Request:    req,
        },
    )
    
    // Schedule initial task
    backfiller.ScheduleTask(BackfillerTask{
        FireTime: time.Now(),
    })
}

Component Deletion

Components can delete themselves:

func (b *Backfiller) OnBackfillerTask() {
    if b.IsComplete() {
        b.DeleteSelf()
        return
    }
    
    // Continue processing
    b.ProcessNextBatch()
}

State Transitions

Components transition state via task handlers:

func (c *Component) OnTimerTask() {
    // 1. Read current state
    state := c.GetState()
    
    // 2. Apply transition logic
    state.Counter++
    state.LastUpdate = time.Now()
    
    // 3. Schedule next task
    if !state.IsComplete {
        c.ScheduleTask(TimerTask{
            FireTime: time.Now().Add(time.Minute),
        })
    }
    
    // 4. Persist state
    c.UpdateState(state)
}

Request-Reply Pattern

Chasm supports synchronous request-reply interactions:

Request Types

Queries: Read-only, do not modify state
Updates: Modify state, wait for effect to be durable
Signals: Fire-and-forget state modifications

Request Handling

Components implement request handlers:

type SchedulerComponent interface {
    // Query handler (read-only)
    OnDescribeRequest(req DescribeRequest) DescribeResponse
    
    // Update handler (modifies state)
    OnPauseRequest(req PauseRequest) PauseResponse
    
    // Signal handler (fire-and-forget)
    OnTriggerSignal(signal TriggerSignal)
}

Routing

Requests are routed to the appropriate component:

Root requests: Sent to root component
Component-specific requests: Include ComponentPath in request
Broadcast requests: Fan out to multiple components

Example: Scheduler Implementation

Temporal’s Scheduler feature is implemented using Chasm:

Component Structure

Task Flow

Generator Task:

Fires periodically based on schedule spec
Buffers actions into Invoker
Reschedules itself for next interval

Invoker Tasks:

ProcessBufferTask: Evaluates overlap policies
ExecuteTask: Starts workflows via RPC

Backfiller Task:

Processes manual backfill requests
Buffers actions into Invoker
Deletes itself when complete

See Schedules Architecture for full details.

Completion Callbacks

Chasm supports asynchronous completion callbacks for side effects:

Nexus Completion

When a component starts a workflow:

Generate a completion callback token
Include token in StartWorkflowExecution request
When workflow completes, callback is triggered
Component receives notification and updates state

// Start workflow with callback
token := chasm.GenerateCallbackToken(componentRef)
resp, err := client.StartWorkflowExecution(ctx, &workflowservice.StartWorkflowExecutionRequest{
    // ... other fields
    CompletionCallbacks: []*commonpb.Callback{
        {
            Variant: &commonpb.Callback_Nexus_{
                Nexus: &commonpb.Callback_Nexus{
                    Url: callbackURL,
                    Token: token,
                },
            },
        },
    },
})

// Later, when workflow completes
func (c *Component) OnCompletionCallback(result NexusResult) {
    c.state.LastResult = result
    c.UpdateState(c.state)
}

Advantages of Chasm

Composition

Build complex features by composing simpler components hierarchically.

Reusability

Components can be reused across different archetypes.

Testability

Components can be tested independently.

Evolution

Components can be added/removed without breaking existing state.

Performance

Only active components are loaded into memory.

Comparison with Workflows

Aspect	Traditional Workflow	Chasm Execution
Structure	Single state machine	Tree of state machines
State	Workflow variables	Per-component state
Tasks	Workflow & Activity Tasks	Component-specific tasks
Composition	Child workflows	Component hierarchy
Lifecycle	Start → Complete	Dynamic component creation/deletion
Use Cases	Business workflows	Advanced Temporal features

Implementation Details

Storage

Chasm state is stored in the same tables as workflow state:

Component state in Mutable State columns
Component tasks in History task queues
Uses existing persistence infrastructure

Code Organization

Chasm library is located in /chasm directory:

/chasm/lib: Component implementations (Scheduler, Activity, NexusOperation)
/chasm/interfaces.go: Core interfaces
/chasm/nexus_completion.go: Completion callback support

// Core Chasm interfaces
type Component interface {
    // State management
    GetState() interface{}
    UpdateState(state interface{}) error
    
    // Task scheduling
    ScheduleTask(task Task)
    
    // Child management
    CreateChildComponent(componentType string, state interface{}) Component
    DeleteSelf()
    
    // Request handling
    HandleRequest(req Request) (Response, error)
}

Integration with History Service

Chasm components are executed by History Service:

Task execution goes through existing queue processors
State updates use existing transaction mechanisms
No separate service required

// Code entrypoint: service/history/chasm_engine.go
// ChasmEngine handles Chasm component execution

Use Cases

Chasm is used to implement:

Scheduler

Periodic workflow execution with backfill support

Worker Deployment

Managing worker versioning and deployment

Nexus Operations

Cross-namespace workflow invocations

Activity Execution

(Planned) Enhanced activity state management

Development Guidelines

When to Use Chasm

Consider Chasm when:

Feature requires multiple concurrent state machines
Need dynamic component creation/deletion
Require sophisticated lifecycle management
State machine structure evolves over time

When NOT to Use Chasm

Don’t use Chasm when:

Simple workflow is sufficient
State machine structure is fixed
No need for hierarchical composition
Complexity overhead not justified

Best Practices

Keep Components Focused

Each component should have a single, well-defined responsibility.

Minimize Component State

Store only essential state; derive everything else.

Handle Failures Gracefully

Tasks may fail and be retried; ensure idempotency.

Document State Machines

Use diagrams to document component interactions and state transitions.

Future Directions

Chasm is evolving to support:

More archetypes: Activity, Query, Update as first-class Chasm components
Better tooling: Visualization and debugging tools
Cross-cluster support: Enhanced multi-cluster coordination
Performance optimizations: Lazy loading, state compression

Chasm is an internal framework and its APIs may change. External developers should not depend on Chasm directly but use the stable public APIs that are built on top of it.

Schedules Architecture

Detailed Scheduler implementation using Chasm

History Service

How Chasm integrates with History Service

Event Sourcing

State management foundation

Architecture Overview

High-level system architecture

Getting Started

Core Concepts

Deployment

Architecture

Operations

Advanced Features

​Chasm Architecture

​Motivation

​Core Concepts

​Terminology

​Architecture

​Hierarchical State Machines

​State Management

​Task Processing

​Component Lifecycle

​Component Creation

​Component Deletion

​State Transitions

​Request-Reply Pattern

​Request Types

​Request Handling

​Routing

​Example: Scheduler Implementation

​Component Structure

​Task Flow

​Completion Callbacks

​Nexus Completion

​Advantages of Chasm

​Comparison with Workflows

​Implementation Details

​Storage

​Code Organization

​Integration with History Service

​Use Cases

Scheduler

Worker Deployment

Nexus Operations

Activity Execution

​Development Guidelines

​When to Use Chasm

​When NOT to Use Chasm

​Best Practices

​Future Directions

​Further Reading

Schedules Architecture

History Service

Event Sourcing

Architecture Overview

Build docs developers (and LLMs) love

Chasm Architecture

Motivation

Core Concepts

Terminology

Architecture

Hierarchical State Machines

State Management

Task Processing

Component Lifecycle

Component Creation

Component Deletion

State Transitions

Request-Reply Pattern

Request Types

Request Handling

Routing

Example: Scheduler Implementation

Component Structure

Task Flow

Completion Callbacks

Nexus Completion

Advantages of Chasm

Comparison with Workflows

Implementation Details

Storage

Code Organization

Integration with History Service

Use Cases

Development Guidelines

When to Use Chasm

When NOT to Use Chasm

Best Practices

Future Directions

Further Reading