Retry Policies

Overview

Temporal automatically retries failed activities and child workflows according to configurable retry policies:

Exponential backoff - Increasing delays between retry attempts
Maximum attempts - Limit total retry count
Timeout controls - Cap retry duration
Non-retriable errors - Fail immediately for specific error types
Backoff coefficient - Control retry delay growth rate

Retry policies are core to Temporal’s reliability guarantees. Configure them carefully to balance fault tolerance with resource usage.

Retry Policy Configuration

initialInterval

duration

required

Delay before the first retry attemptDefault: 1 second

InitialInterval: time.Second

backoffCoefficient

float

required

Multiplier for retry delay on each attemptDefault: 2.0

BackoffCoefficient: 2.0 // Doubles delay each retry

maximumInterval

duration

Maximum delay between retriesDefault: 100x initial interval

MaximumInterval: time.Minute * 5

maximumAttempts

int

Maximum number of retry attempts (0 = unlimited)Default: 0 (unlimited)

MaximumAttempts: 10

nonRetriableErrorTypes

[]string

Error types that should not be retried

NonRetriableErrorTypes: []string{"ValidationError", "PermissionDenied"}

Exponential Backoff Formula

Retry delay calculation:

retryDelay = min(
    initialInterval * (backoffCoefficient ^ attemptNumber),
    maximumInterval
)

Example: Default Policy

With default settings (initialInterval=1s, backoffCoefficient=2.0, maximumInterval=100s):

Attempt	Delay Calculation	Actual Delay
1	1s × 2^0 = 1s	1s
2	1s × 2^1 = 2s	2s
3	1s × 2^2 = 4s	4s
4	1s × 2^3 = 8s	8s
5	1s × 2^4 = 16s	16s
6	1s × 2^5 = 32s	32s
7	1s × 2^6 = 64s	64s
8	1s × 2^7 = 128s	100s (capped)
9	1s × 2^8 = 256s	100s (capped)

Activity Retry Policies

Configure retries when scheduling activities:

Go
TypeScript
Python

ao := workflow.ActivityOptions{
    StartToCloseTimeout: time.Minute * 10,
    RetryPolicy: &temporal.RetryPolicy{
        InitialInterval:    time.Second,
        BackoffCoefficient: 2.0,
        MaximumInterval:    time.Minute,
        MaximumAttempts:    5,
        NonRetriableErrorTypes: []string{
            "ValidationError",
        },
    },
}
ctx = workflow.WithActivityOptions(ctx, ao)

err := workflow.ExecuteActivity(ctx, MyActivity, input).Get(ctx, &result)

const result = await proxyActivities<typeof activities>({
  startToCloseTimeout: '10m',
  retry: {
    initialInterval: '1s',
    backoffCoefficient: 2.0,
    maximumInterval: '1m',
    maximumAttempts: 5,
    nonRetriableErrorTypes: ['ValidationError'],
  },
}).myActivity(input);

result = await workflow.execute_activity(
    my_activity,
    input,
    start_to_close_timeout=timedelta(minutes=10),
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=1),
        backoff_coefficient=2.0,
        maximum_interval=timedelta(minutes=1),
        maximum_attempts=5,
        non_retriable_error_types=["ValidationError"],
    ),
)

Workflow Retry Policies

Configure retries for workflows and child workflows:

// Start workflow with retry policy
workflowOptions := client.StartWorkflowOptions{
    ID:        "my-workflow-id",
    TaskQueue: "my-task-queue",
    RetryPolicy: &temporal.RetryPolicy{
        InitialInterval:    time.Second * 5,
        BackoffCoefficient: 2.0,
        MaximumInterval:    time.Hour,
        MaximumAttempts:    3,
    },
}

we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)

Workflow retries create a new workflow execution run. The workflow will restart from the beginning on each retry.

Server-Side Configuration

Default retry policies can be configured in dynamic config:

# Dynamic config for default activity retry
frontend.defaultActivityRetryPolicy:
  InitialIntervalInSeconds: 1
  BackoffCoefficient: 2.0
  MaximumIntervalInSeconds: 100
  MaximumAttempts: 0

# Dynamic config for default workflow retry  
frontend.defaultWorkflowRetryPolicy:
  InitialIntervalInSeconds: 1
  BackoffCoefficient: 2.0
  MaximumIntervalInSeconds: 100
  MaximumAttempts: 0

Non-Retriable Errors

Specify error types that should fail immediately without retry:

type ValidationError struct {
    message string
}

func (e *ValidationError) Error() string {
    return e.message
}

// Configure in retry policy
RetryPolicy: &temporal.RetryPolicy{
    NonRetriableErrorTypes: []string{
        "ValidationError",      // Match by type name
        "*errors.errorString",  // Match wrapped errors
    },
}

Error Matching

Error type matching rules:

Exact type name match - "ValidationError" matches ValidationError type
Package path match - "myapp/errors.ValidationError" matches fully qualified type
Wildcard match - "*ValidationError" matches any package

Implementation in History Service

Non-retriable error checking is implemented in service/history/workflow/activity.go:

func isRetriable(err error, policy *RetryPolicy) bool {
    errorType := reflect.TypeOf(err).String()
    for _, nonRetriable := range policy.NonRetriableErrorTypes {
        if strings.HasSuffix(errorType, nonRetriable) {
            return false
        }
    }
    return true
}

Retry Timeouts

Retries interact with activity and workflow timeouts:

Activity Timeouts

ScheduleToStart

Time from schedule to worker pickup. Retries reset this timeout.

StartToClose

Time for single attempt. Each retry gets full StartToClose timeout.

ScheduleToClose

Total time including all retries. Stops retry loop when exceeded.

Heartbeat

Heartbeat timeout. Retries reset on heartbeat timeout.

Retry Until ScheduleToClose

ao := workflow.ActivityOptions{
    StartToCloseTimeout:    time.Minute,      // 1 minute per attempt
    ScheduleToCloseTimeout: time.Minute * 10, // 10 minutes total
    RetryPolicy: &temporal.RetryPolicy{
        InitialInterval: time.Second,
        MaximumAttempts: 0, // Unlimited
    },
}

Activity will retry until:

ScheduleToCloseTimeout (10 minutes) is exceeded
OR activity succeeds
OR non-retriable error occurs

Best Practices

Use appropriate backoff for service dependencies

Match backoff to downstream service recovery time:

Fast recovery (seconds): initialInterval=1s, backoffCoefficient=1.5
Slow recovery (minutes): initialInterval=30s, backoffCoefficient=2.0
Rate limited APIs: initialInterval=5s, maximumInterval=5m

Limit retries for expensive operations

Set maximumAttempts for costly activities:

RetryPolicy: &temporal.RetryPolicy{
    InitialInterval: time.Second * 5,
    MaximumAttempts: 3, // Only retry twice
}

Mark validation errors as non-retriable

Don’t waste resources retrying permanent failures:

NonRetriableErrorTypes: []string{
    "ValidationError",
    "AuthenticationError",
    "AuthorizationError",
    "NotFoundError",
}

Use ScheduleToClose to bound total retry time

Prevent infinite retry loops:

ao := workflow.ActivityOptions{
    ScheduleToCloseTimeout: time.Hour, // Max 1 hour of retries
    RetryPolicy: &temporal.RetryPolicy{
        MaximumAttempts: 0, // Unlimited attempts within 1 hour
    },
}

Monitoring Retry Behavior

Key metrics for retry monitoring:

activity_task_schedule_to_start_latency - Time waiting for worker (increases with retries)
activity_execution_failed - Failed activity attempts (includes retries)
activity_execution_failed_total - Activities failed after all retries
activity_retry_count - Number of retry attempts per activity

Implementation Details

Retry logic is implemented in:

Activity retries: service/history/workflow/activity.go
Workflow retries: service/history/workflow/mutable_state_impl.go
Backoff calculation: common/backoff/retry.go
Timer scheduling: History Service timer queue

Retry state is persisted in workflow mutable state:

type ActivityInfo struct {
    Attempt          int32
    LastFailure      *failurepb.Failure
    ScheduledTime    *timestamppb.Timestamp
    RetryLastWorkerIdentity string
    // ...
}

Activities

Learn about activity execution and timeouts

Workflows

Understand workflow execution model

History Service

Deep dive into retry implementation

Getting Started

Core Concepts

Deployment

Architecture

Operations

Advanced Features

Overview

Retry Policy Configuration

Exponential Backoff Formula

Example: Default Policy

Activity Retry Policies

Workflow Retry Policies

Server-Side Configuration

Non-Retriable Errors

Error Matching

Retry Timeouts

Activity Timeouts

ScheduleToStart

StartToClose

ScheduleToClose

Heartbeat

Retry Until ScheduleToClose

Best Practices

Monitoring Retry Behavior

Implementation Details

See Also

Activities

Workflows

History Service

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Deployment

Architecture

Operations

Advanced Features

​Overview

​Retry Policy Configuration

​Exponential Backoff Formula

​Example: Default Policy

​Activity Retry Policies

​Workflow Retry Policies

​Server-Side Configuration

​Non-Retriable Errors

​Error Matching

​Retry Timeouts

​Activity Timeouts

ScheduleToStart

StartToClose

ScheduleToClose

Heartbeat

​Retry Until ScheduleToClose

​Best Practices

​Monitoring Retry Behavior

​Implementation Details

​See Also

Activities

Workflows

History Service

Build docs developers (and LLMs) love

Overview

Retry Policy Configuration

Exponential Backoff Formula

Example: Default Policy

Activity Retry Policies

Workflow Retry Policies

Server-Side Configuration

Non-Retriable Errors

Error Matching

Retry Timeouts

Activity Timeouts

Retry Until ScheduleToClose

Best Practices

Monitoring Retry Behavior

Implementation Details

See Also