Skip to main content

​
Workflows

Workflows are the core abstraction in Temporal Server that enable durable execution of application logic. A Workflow Execution represents a single run of a workflow definition, tracked through an append-only event history that enables fault-tolerant, resumable execution.

​
Event Sourcing Model

Temporal implements workflow durability through event sourcing. Every workflow execution maintains a complete history of events that describe everything that has happened during its execution.

​
Workflow History

The Workflow History is a linear sequence of History Events stored durably in the persistence layer. Each event type represents a specific state transition:

WorkflowExecutionStarted

Marks the beginning of a workflow execution with input parameters and configuration

WorkflowTaskScheduled

Indicates a workflow task has been created and is ready for worker pickup

ActivityTaskScheduled

Records that an activity has been scheduled for execution

TimerFired

Captures when a workflow timer expires

​
Deterministic Replay

Workflow code must be deterministic because execution state is recovered by replaying the history:
  1. When a worker picks up a workflow task, it receives the complete event history
  2. The SDK replays events through the workflow code to reconstruct state
  3. Workflow continues execution from where it left off
  4. New commands are generated and sent back to the server
// From service/history/workflow/mutable_state_impl.go
type MutableStateImpl struct {
    pendingActivityInfoIDs     map[int64]*persistencespb.ActivityInfo
    pendingTimerInfoIDs        map[string]*persistencespb.TimerInfo
    pendingChildExecutionInfoIDs map[int64]*persistencespb.ChildExecutionInfo
    
    executionInfo  *persistencespb.WorkflowExecutionInfo
    executionState *persistencespb.WorkflowExecutionState
    
    hBuilder *historybuilder.HistoryBuilder
    // ... workflow state tracking
}

​
Mutable State

For performance, Temporal maintains a cached summary of workflow state called Mutable State rather than recomputing everything from history on each access.

​
What Mutable State Tracks

Mutable State includes:
  • Pending Activities: Activities that are scheduled or running
  • Pending Timers: Active timers set by the workflow
  • Pending Child Workflows: Child workflow executions in progress
  • Pending Signals: External signal information
  • Execution Info: Workflow type, task queue, timeouts
  • Execution State: Current status (Running, Completed, Failed, etc.)
// Mutable State contains in-memory summaries of workflow state
pendingActivityTimerHeartbeats map[int64]time.Time
pendingActivityInfoIDs         map[int64]*persistencespb.ActivityInfo
pendingTimerInfoIDs           map[string]*persistencespb.TimerInfo

// Modified data since last persistence write
updateActivityInfos map[int64]*persistencespb.ActivityInfo
deleteActivityInfos map[int64]struct{}
Mutable State is persisted alongside history events and cached in memory for recently accessed workflows.

​
Workflow Lifecycle

A typical workflow execution follows this sequence:

​
State Transitions

Each RPC or timer expiration triggers a state transition:
  1. Input: RPC from application, worker response, or timer fired
  2. Processing:
    • Create new History Events
    • Update Mutable State
    • Create History Tasks (Transfer or Timer tasks)
  3. Output: Atomic transaction persisting events, state, and tasks
// From service/history/api/update_workflow_util.go
// State transitions use a common code path:
GetAndUpdateWorkflowWithNew(
    ctx,
    workflowContext,
    updateFunc, // Function that modifies mutable state
)
// Performs atomic write of:
// - New history events
// - Updated mutable state  
// - Generated tasks

​
Workflow Tasks

A Workflow Task is how the History Service prompts a worker to advance workflow execution.

​
When Workflow Tasks Are Created

  • Workflow execution starts
  • Activity completes (success or failure)
  • Timer fires
  • External signal received
  • Query received

​
Workflow Task Processing

1

Worker polls and receives task

Worker makes long-poll request to Matching Service, which returns a workflow task containing the execution’s event history
2

SDK replays history through workflow code

The SDK feeds history events through deterministic workflow code to reconstruct state and determine next actions
3

Worker sends commands back

Worker returns a list of commands like ScheduleActivityTask, StartTimer, CompleteWorkflowExecution
4

History Service processes commands

Commands are converted to new history events and state updates in an atomic transaction

​
Workflow Sharding

Workflow executions are distributed across History Shards for scalability:
  • Each shard is a logical partition managing a subset of workflows
  • Shard count is fixed at cluster creation (typically 512-16,384 shards)
  • Each History Service instance owns multiple shards
  • Shard ownership uses membership protocol (Ringpop) for distribution
Workflows are assigned to shards by hashing the Namespace ID and Workflow ID:
// Workflow is routed to specific shard
shardID := hash(namespaceID, workflowID) % numShards
Each shard independently:
  • Handles RPCs for its workflows
  • Processes background tasks
  • Maintains its own task queues
  • Has isolated failure domain

​
Consistency Guarantees

Temporal provides strong consistency for workflow state:

​
Within History Service

  • Mutable State + History Events: Consistent via event sourcing; mutable state tracks which event was last processed
  • Mutable State + History Tasks: Consistent via database transactions
  • History to Matching: Eventually consistent via Transfer Task queue and Transactional Outbox pattern

​
Failure Handling

Process Crash

Workflow state is durable in persistence. Another History instance takes over the shard and continues processing.

Task Failure

Tasks in Transfer/Timer queues are retried. Ack levels ensure tasks aren’t lost during failures.

Network Partition

Shard ownership fencing via RangeID prevents split-brain. Only one instance can write to a shard.

Database Unavailable

Requests are retried with backoff. In-memory state is reloaded from persistence when available.

​
Best Practices

Workflow code must be deterministic. Side effects (I/O, randomness, system time) must go through Activities or SDK APIs that are recorded in history.
Keep workflow history bounded. Very long-running workflows with millions of events can impact performance. Consider Continue-As-New for indefinite workflows.

Build docs developers (and LLMs) love

Get started for freeTalk to us