Skip to main content

​
History Service

The History Service is the core of Temporal’s durable execution engine. It manages individual Workflow Executions, maintains their state, and drives them to completion by coordinating with the Matching Service and Worker processes.

​
Overview

The History Service handles two main types of gRPC requests:
  1. Requests from User Applications: Start, Cancel, Query, Update, Signal, Reset, etc.
  2. Requests from Workers: Completion of Workflow Tasks and Activity Tasks
For system architecture context, see the Architecture Overview

​
Core Responsibilities

When handling requests, the History Service:
  1. Determines new History Events that the request implies and appends them to Workflow History
  2. Creates Transfer Tasks if Workflow or Activity Tasks should be added to Matching Service
  3. Creates Timer Tasks when the workflow is blocked on a timer or timeout
  4. Updates Mutable State to reflect the current state of the workflow execution

​
History Shards

​
Sharding Architecture

A single Temporal Cluster manages millions of concurrent Workflow Executions by partitioning them into History Shards:
  • The total number of shards is fixed at cluster creation and cannot be changed
  • Each workflow execution is assigned to exactly one shard (based on workflow ID hash)
  • Multiple History Service instances run in the cluster, each owning a subset of shards
  • Shard ownership is coordinated via the Ringpop membership protocol

​
Shard Ownership

“Owning” a History Shard means:
  • Synchronously handling all incoming RPC requests for workflows in that shard
  • Asynchronously processing background tasks (timers, transfers, visibility, replication)
  • Maintaining in-memory metadata including:
    • RangeID: Monotonically increasing generation number for fencing
    • Queue states: Acknowledged/processed positions of each internal queue

​
Persistence Mapping

With Cassandra, History Shards map 1:1 to partitions of the executions table. Similar mappings exist for MySQL, PostgreSQL, and SQLite. 
// Code entrypoint: service/history/history_engine.go
// When History service starts, Start() is called for each shard owned by that host

​
Workflow Execution Mutable State

​
What is Mutable State?

For every workflow execution, Temporal maintains a collection of data structures called Mutable State that summarizes:
  • Identities of in-progress activities
  • Active timers and their expiration times
  • Child workflow executions
  • Pending signals and updates
  • Current workflow task information
  • Execution status and result

​
Why Cache Mutable State?

While this data could be recomputed from Workflow History Events, doing so would be slow. Therefore:
  • Mutable State is persisted alongside history events
  • Recently accessed executions are cached in memory
  • Cache improves latency for active workflows

​
Persistence Details

Due to Cassandra’s limited RDBMS support (the most important persistence backend), Mutable State is persisted in a single row with a layout similar to its in-memory representation. 
// Code entrypoint: service/history/workflow/mutable_state_impl.go
// MutableStateImpl implements the MutableState interface
// Loaded from persistence: service/history/workflow/context.go:248

​
Workflow Execution History

​
Event Structure

Workflow Execution History is a linear sequence of History Events (or a branching topology for reset workflows):
  • Each event has an EventID (monotonically increasing)
  • Some events carry payloads (e.g., activity results, signal data)
  • Events are immutable once written
  • Events are publicly exposed via API

​
Event Sourcing Property

The sequence of History Events alone is sufficient to recover:
  • All Mutable State information
  • All tasks relating to the workflow execution
This is the foundation of Temporal’s event sourcing implementation.

​
Event Types

Events represent state transitions in workflow execution:
  • WorkflowExecutionStarted
  • WorkflowTaskScheduled, WorkflowTaskStarted, WorkflowTaskCompleted
  • ActivityTaskScheduled, ActivityTaskStarted, ActivityTaskCompleted
  • TimerStarted, TimerFired
  • WorkflowExecutionCompleted, WorkflowExecutionFailed
  • And many more…
See temporal/api/enums/v1/event_type.proto for the complete list.
In Temporal, “event” refers to Workflow History Event, not event-driven architecture events.

​
RPC Handling

​
User Application Requests

Handlers for user application requests are defined in service/history/api/:
  • StartWorkflow, SignalWorkflow, UpdateWorkflow
  • CancelWorkflow, TerminateWorkflow
  • QueryWorkflow, DescribeWorkflow
  • ResetWorkflow
Many of these use a shared code path for state transitions. 
// Code entrypoint: service/history/api/signalworkflow/api.go
// Example: SignalWorkflow request handling

​
Worker Completion Requests

Handlers for worker completion requests: RespondWorkflowTaskCompleted RespondActivityTaskCompleted
  • Contains the result of the activity
Failure Responses
  • RespondWorkflowTaskFailed, RespondActivityTaskFailed, etc.
// Code entrypoint: service/history/workflow_task_handler_callbacks.go:371
// WorkflowTaskCompleted handler accumulates updates from commands

​
Queue Processing

The “task queues” discussed here are internal History Service implementation details, distinct from the user-facing Task Queues in Matching Service.

​
Queue Types

Each History Shard manages several internal task queues:
  1. Transfer Queue: Immediate execution tasks
  2. Timer Queue: Scheduled execution tasks
  3. Visibility Queue: Update visibility store
  4. Replication Queue: Cross-cluster replication
  5. Archival Queue: Archive history to long-term storage

​
Queue Processor Architecture

When the History Service starts, for each owned shard:
  1. A QueueProcessor starts for each queue type
  2. Multiple Reader goroutines read tasks from persistence
  3. Tasks are submitted to a shared task execution framework
  4. Goroutine pool executes tasks across all shards
  5. Queue “ack levels” are periodically checkpointed
// Code entrypoint: service/history/history_engine.go:288
// Start() initiates queue processors for each shard
// Readers: service/history/queues/reader.go:51
// Task framework: common/tasks/

​
Transfer Queue

Transfer Tasks are available for immediate execution:
  • Enqueue Workflow Task in Matching Service
  • Enqueue Activity Task in Matching Service
  • Start Child Workflow Execution
  • Cancel External Workflow Execution
  • Signal External Workflow Execution
  • Delete Workflow Execution
When executed, a Transfer Task typically makes an RPC to the Matching Service. 
// Code entrypoint: service/history/transfer_queue_active_task_executor.go:114
// Execute() method dispatches by task type
// processActivityTask: transfer_queue_active_task_executor.go:169
// processWorkflowTask: transfer_queue_active_task_executor.go:217

​
Timer Queue

Timer Tasks become available at their trigger time:
  • User Timer Timeout: workflow.sleep() timer expires
  • Activity Timeout: Schedule-to-start, start-to-close, heartbeat timeouts
  • Workflow Task Timeout: Workflow task not completed in time
  • Workflow Timeout: Workflow execution timeout
  • Delete History Event: Retention period expired
When a user timer fires, a new Workflow Task is scheduled to advance the workflow. 
// Code entrypoint: service/history/timer_queue_active_task_executor.go:90
// Execute() handles tasks differently by timer type
// executeUserTimerTimeoutTask: timer_queue_active_task_executor.go:136
// Activity retries: timer_queue_active_task_executor.go:426

​
Visibility Queue

Visibility Tasks update workflow metadata in the visibility storage:
  • StartWorkflowExecution: Record new workflow
  • CloseWorkflowExecution: Record workflow completion
  • UpsertWorkflowSearchAttributes: Update search attributes
Visibility storage may be a different database than the primary persistence store.

​
State Transitions

​
State Transition Pattern

A workflow execution state transition follows this pattern: All state transitions use the same implementation pathway:
  1. In-memory: Create new History Event(s), update Mutable State, create History Tasks
  2. Persistence: Append History Events to storage
  3. Atomically: Update Mutable State and add History Tasks in a database transaction
// Code entrypoint: service/history/api/update_workflow_util.go:37
// GetAndUpdateWorkflowWithNew is the generic utility for state transitions
// Calls: service/history/workflow/context.go:459 (UpdateWorkflowExecutionAsActive)
// SQL persistence: common/persistence/sql/execution.go:339
// Cassandra persistence: common/persistence/cassandra/execution_store.go:142

​
State Transition Call Sites

RPC from User Application: 
// Example: SignalWorkflow
// service/history/api/signalworkflow/api.go:54
RPC from Worker: 
// Example: WorkflowTaskCompleted
// service/history/workflow_task_handler_callbacks.go:690
Timer Task Processor: 
// Example: User timer timeout
// service/history/timer_queue_active_task_executor.go:136

​
Consistency Guarantees

Temporal maintains strong consistency through:
Mutable State and History Tasks are persisted atomically using database transactions, ensuring they are always consistent with each other.
Mutable State stores the identity of the latest History Event. An event is only “valid” if it’s reflected in Mutable State. On failure to persist, we reload from persistence.
Transfer Tasks written by a History Shard will eventually result in the required Workflow or Activity Task being created in Matching Service. This is the Transactional Outbox pattern.

​
Example: Simple Workflow Execution

Here’s how a simple workflow that calls an activity is processed: 
function myWorkflow() {
  result = callActivity(myActivity);
  workflowSleep(THIRTY_DAYS);
  return result;
}
For detailed sequence diagrams, see Workflow Lifecycle.

​
Additional Responsibilities

The History Service has many other responsibilities not covered in detail here:
  • Shard Membership and Ownership: Managing which service instance owns which shards
  • Replication: Cross-cluster replication and conflict resolution
  • Visibility: Updating visibility storage for workflow search
  • Archival: Archiving histories to long-term storage
  • Workflow Reset: Resetting workflow execution to a previous point

​
Further Reading

Workflow Lifecycle

Detailed sequence diagrams of workflow execution

Matching Service

How tasks are distributed to workers

Event Sourcing

How Temporal implements event sourcing

Architecture Overview

High-level system architecture

Build docs developers (and LLMs) love

Get started for freeTalk to us