Metrics Reference - Temporal Server

This page provides a comprehensive reference of all metrics emitted by Temporal Server.

Metric Types

Temporal uses four metric types:

Counter - Monotonically increasing value (e.g., request count)
Gauge - Point-in-time value that can increase or decrease (e.g., queue size)
Timer - Duration measurements with histogram distribution
Histogram - Value distribution (bytes, dimensionless counts)

Common Metric Tags

All metrics may include these tags:

Tag	Description	Values
`operation`	API method or operation name	Method-specific
`service_role`	Service type	`frontend`, `history`, `matching`, `worker`, `admin`
`namespace`	Namespace name	User-defined
`task_category`	Task category type	`transfer`, `timer`, `visibility`, `archival`
`task_type`	Specific task type	Task-specific
`failure`	Failure reason	Error-specific
`error_type`	Error classification	`invalid_argument`, `not_found`, etc.
`cache_type`	Cache identifier	`mutablestate`, `events`, etc.
`db_kind`	Database type	`cassandra`, `mysql`, `postgres`, `sqlite`

Service-Level Metrics

Request Metrics

`service_requests`

Type: Counter
Description: Total number of RPC requests received by the service
Tags: operation, service_role

`service_pending_requests`

Type: Gauge
Description: Current number of pending/in-flight requests
Tags: operation, service_role

`service_latency`

Type: Timer
Description: End-to-end request latency including user code execution
Tags: operation, service_role
Unit: milliseconds

`service_latency_nouserlatency`

Type: Timer
Description: Server-side processing latency excluding user code execution
Tags: operation, service_role
Unit: milliseconds

`service_latency_userlatency`

Type: Timer
Description: Time spent in user code (workflow/activity execution)
Tags: operation, service_role
Unit: milliseconds

Error Metrics

`service_errors`

Type: Counter
Description: Total unexpected service errors
Tags: operation, service_role

`service_error_with_type`

Type: Counter
Description: Service errors categorized by error type
Tags: operation, service_role, error_type

`service_errors_invalid_argument`

Type: Counter
Description: Requests rejected due to invalid arguments
Tags: operation, service_role

`service_errors_namespace_not_active`

Type: Counter
Description: Requests to inactive namespaces
Tags: operation, service_role, namespace

`service_errors_resource_exhausted`

Type: Counter
Description: Requests rejected due to resource limits
Tags: operation, service_role, resource_exhausted_cause, resource_exhausted_scope

`service_errors_entity_not_found`

Type: Counter
Description: Requests for non-existent entities (workflow, namespace, etc.)
Tags: operation, service_role

`service_errors_execution_already_started`

Type: Counter
Description: Attempts to start already-running workflows
Tags: operation, service_role

`service_errors_context_timeout`

Type: Counter
Description: Requests that timed out
Tags: operation, service_role

`service_errors_unauthorized`

Type: Counter
Description: Authorization denied requests
Tags: operation, service_role, namespace

`service_errors_authorize_failed`

Type: Counter
Description: Authorization system failures
Tags: operation, service_role, namespace

Connection Metrics

`service_grpc_conn_accepted`

Type: Counter
Description: Number of gRPC TCP connections accepted
Tags: service_role

`service_grpc_conn_closed`

Type: Counter
Description: Number of gRPC TCP connections closed
Tags: service_role

`service_grpc_conn_active`

Type: Gauge
Description: Current number of active gRPC connections
Tags: service_role

`service_dial_latency`

Type: Timer
Description: Latency of establishing new TCP connections
Tags: service_role
Unit: milliseconds

`service_dial_success`

Type: Counter
Description: Successful TCP connection attempts
Tags: service_role

`service_dial_error`

Type: Counter
Description: Failed TCP connection attempts
Tags: service_role

Persistence Metrics

All persistence metrics follow the pattern: Persistence{Operation}Scope

Shard Operations

`GetOrCreateShard`

Description: Shard metadata retrieval or initialization
Tags: db_kind

`UpdateShard`

Description: Shard metadata updates
Tags: db_kind

`AssertShardOwnership`

Description: Shard ownership validation
Tags: db_kind

Workflow Execution Operations

`CreateWorkflowExecution`

Description: New workflow execution creation
Tags: db_kind

`GetWorkflowExecution`

Description: Workflow execution state retrieval
Tags: db_kind

`UpdateWorkflowExecution`

Description: Workflow execution state updates
Tags: db_kind

`DeleteWorkflowExecution`

Description: Workflow execution deletion
Tags: db_kind

`GetCurrentExecution`

Description: Current workflow run retrieval
Tags: db_kind

History Operations

`AppendHistoryNodes`

Description: History event append operations
Tags: db_kind

`ReadHistoryBranch`

Description: History event reads
Tags: db_kind

`ReadHistoryBranchReverse`

Description: Reverse chronological history reads
Tags: db_kind

`DeleteHistoryBranch`

Description: History branch deletion
Tags: db_kind

`ForkHistoryBranch`

Description: History branch creation for reset
Tags: db_kind

Task Queue Operations

`GetTransferTasks`

Description: Transfer task retrieval
Tags: db_kind

`CompleteTransferTask`

Description: Transfer task completion
Tags: db_kind

`GetTimerTasks`

Description: Timer task retrieval
Tags: db_kind

`CompleteTimerTask`

Description: Timer task completion
Tags: db_kind

`GetVisibilityTasks`

Description: Visibility task retrieval
Tags: db_kind

`GetReplicationTasks`

Description: Replication task retrieval
Tags: db_kind

DLQ Operations

`EnqueueMessageToDLQ`

Description: Messages moved to dead letter queue
Tags: db_kind, queue_type

`ReadMessagesFromDLQ`

Description: DLQ message reads
Tags: db_kind, queue_type

`DeleteMessageFromDLQ`

Description: Individual DLQ message deletion
Tags: db_kind, queue_type

`RangeDeleteMessagesFromDLQ`

Description: Bulk DLQ message deletion
Tags: db_kind, queue_type

`dlq_message_count`

Type: Gauge
Description: Number of messages in DLQ by task category
Tags: task_category
Note: Emitted every 3 hours from shard 1 owner

Namespace Operations

`CreateNamespace`

Description: Namespace creation
Tags: db_kind

`GetNamespace`

Description: Namespace metadata retrieval
Tags: db_kind

`UpdateNamespace`

Description: Namespace metadata updates
Tags: db_kind

`ListNamespaces`

Description: Namespace listing
Tags: db_kind

History Service Metrics

Workflow Operations

`StartWorkflowExecution`

Description: Workflow start requests
Tags: namespace

`RespondWorkflowTaskCompleted`

Description: Workflow task completion processing
Tags: namespace

`RespondActivityTaskCompleted`

Description: Activity task completion processing
Tags: namespace

`RecordActivityTaskHeartbeat`

Description: Activity heartbeat processing
Tags: namespace

Shard Management

`ShardController`

Description: Shard controller operations
Tags: shard-id

`ShardInfo`

Description: Shard metadata operations
Tags: shard-id

Task Processing

`TransferQueueProcessor`

Description: Transfer queue processing
Tags: task_type

`TimerQueueProcessor`

Description: Timer queue processing
Tags: task_type

`VisibilityQueueProcessor`

Description: Visibility queue processing
Tags: task_type

`ArchivalQueueProcessor`

Description: Archival queue processing
Tags: task_type

Cache Metrics

`HistoryCacheGetOrCreate`

Description: History cache operations
Tags: namespace

`EventsCacheGetEvent`

Description: Event cache retrievals
Tags: No additional tags

`EventsCachePutEvent`

Description: Event cache insertions
Tags: No additional tags

`EventsCacheGetFromStore`

Description: Event cache misses requiring database fetch
Tags: No additional tags

Matching Service Metrics

`PollWorkflowTaskQueue`

Description: Workflow task poll operations
Tags: namespace

`PollActivityTaskQueue`

Description: Activity task poll operations
Tags: namespace

`AddActivityTask`

Description: Activity task additions to queue
Tags: namespace

`AddWorkflowTask`

Description: Workflow task additions to queue
Tags: namespace

`TaskQueueMgr`

Description: Task queue manager operations
Tags: namespace

Worker Service Metrics

`HistoryArchiver`

Description: History archival operations
Tags: namespace

`VisibilityArchiver`

Description: Visibility archival operations
Tags: namespace

`Replicator`

Description: Cross-cluster replication operations
Tags: namespace, cluster

Authorization Metrics

`service_authorization_latency`

Type: Timer
Description: Authorization check duration
Tags: operation, namespace
Unit: milliseconds

Resource Metrics

Lock Metrics

`lock_requests`

Type: Counter
Description: Lock acquisition attempts
Tags: No additional tags

`lock_latency`

Type: Timer
Description: Time waiting for lock acquisition
Tags: No additional tags
Unit: milliseconds

Semaphore Metrics

`semaphore_requests`

Type: Counter
Description: Semaphore acquisition attempts
Tags: No additional tags

`semaphore_failures`

Type: Counter
Description: Failed semaphore acquisitions
Tags: No additional tags

`semaphore_latency`

Type: Timer
Description: Time waiting for semaphore acquisition
Tags: No additional tags
Unit: milliseconds

TLS Metrics

`certificates_expired`

Type: Gauge
Description: Number of expired TLS certificates
Tags: No additional tags

`certificates_expiring`

Type: Gauge
Description: Number of certificates expiring within warning window
Tags: No additional tags

Workflow Metrics

`state_transition_count`

Type: Histogram
Description: Number of state transitions per workflow task
Tags: namespace

`history_size`

Type: Histogram
Description: Workflow history size in bytes
Tags: namespace
Unit: bytes

`history_count`

Type: Histogram
Description: Number of events in workflow history
Tags: namespace

`event_blob_size`

Type: Histogram
Description: Size of individual event blobs
Tags: namespace
Unit: bytes

`search_attributes_size`

Type: Histogram
Description: Size of search attributes payload
Tags: namespace
Unit: bytes

`memo_size`

Type: Histogram
Description: Size of workflow memo
Tags: namespace
Unit: bytes

Limit Exceeded Metrics

`wf_too_many_pending_activities`

Type: Counter
Description: Workflow tasks failed due to pending activity limit
Tags: namespace

`wf_too_many_pending_child_workflows`

Type: Counter
Description: Workflow tasks failed due to pending child workflow limit
Tags: namespace

`wf_too_many_pending_cancel_requests`

Type: Counter
Description: Workflow tasks failed due to pending cancel request limit
Tags: namespace

`blob_size_error`

Type: Counter
Description: Requests failed due to blob size limits
Tags: namespace

Client Metrics

`client_requests`

Type: Counter
Description: Outbound client requests to other services
Tags: service_role, operation

`client_errors`

Type: Counter
Description: Client request errors
Tags: service_role, operation

`client_latency`

Type: Timer
Description: Client request latency
Tags: service_role, operation
Unit: milliseconds

Metric Collection Best Practices

Histogram Bucket Configuration

Customize histogram buckets based on your workload:

global:
  metrics:
    clientConfig:
      perUnitHistogramBoundaries:
        # For high-throughput, low-latency workflows
        milliseconds: [1, 2, 5, 10, 25, 50, 100, 250, 500, 1000]
        # For large payload workflows
        bytes: [1024, 10240, 102400, 1048576, 10485760]

Tag Cardinality

Be cautious with high-cardinality tags:

global:
  metrics:
    clientConfig:
      excludeTags:
        namespace:
          - "whitelisted-namespace-1"
          - "whitelisted-namespace-2"
        workflowType:
          - "important-workflow"

Sampling for High Volume

For extremely high request rates, consider sampling at the metrics backend level rather than in Temporal.

Getting Started

Core Concepts

Deployment

Architecture

Operations

Advanced Features

​Metric Types

​Common Metric Tags

​Service-Level Metrics

​Request Metrics

​service_requests

​service_pending_requests

​service_latency

​service_latency_nouserlatency

​service_latency_userlatency

​Error Metrics

​service_errors

​service_error_with_type

​service_errors_invalid_argument

​service_errors_namespace_not_active

​service_errors_resource_exhausted

​service_errors_entity_not_found

​service_errors_execution_already_started

​service_errors_context_timeout

​service_errors_unauthorized

​service_errors_authorize_failed

​Connection Metrics

​service_grpc_conn_accepted

​service_grpc_conn_closed

​service_grpc_conn_active

​service_dial_latency

​service_dial_success

​service_dial_error

​Persistence Metrics

​Shard Operations

​GetOrCreateShard

​UpdateShard

​AssertShardOwnership

​Workflow Execution Operations

​CreateWorkflowExecution

​GetWorkflowExecution

​UpdateWorkflowExecution

​DeleteWorkflowExecution

​GetCurrentExecution

​History Operations

​AppendHistoryNodes

​ReadHistoryBranch

​ReadHistoryBranchReverse

​DeleteHistoryBranch

​ForkHistoryBranch

​Task Queue Operations

​GetTransferTasks

​CompleteTransferTask

​GetTimerTasks

​CompleteTimerTask

​GetVisibilityTasks

​GetReplicationTasks

​DLQ Operations

​EnqueueMessageToDLQ

​ReadMessagesFromDLQ

​DeleteMessageFromDLQ

​RangeDeleteMessagesFromDLQ

​dlq_message_count

​Namespace Operations

​CreateNamespace

​GetNamespace

​UpdateNamespace

​ListNamespaces

​History Service Metrics

​Workflow Operations

​StartWorkflowExecution

​RespondWorkflowTaskCompleted

​RespondActivityTaskCompleted

​RecordActivityTaskHeartbeat

​Shard Management

​ShardController

​ShardInfo

​Task Processing

​TransferQueueProcessor

​TimerQueueProcessor