Archival Configuration - Temporal Server

Temporal supports archiving workflow histories and visibility records to external storage for compliance, auditing, and long-term retention.

Overview

Archival moves data from the active persistence store to cheaper, long-term storage:

History Archival - Complete workflow execution history
Visibility Archival - Workflow metadata for search and list

Archival happens automatically after workflow completion based on namespace configuration.

Supported Providers

Temporal includes built-in archival providers:

Filestore - Local or network filesystem
S3 - Amazon S3 and compatible services
Google Cloud Storage - GCS buckets

Global Configuration

Enable archival cluster-wide:

archival:
  history:
    state: "enabled"  # enabled, disabled
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"
      s3store:
        region: "us-east-1"
        endpoint: ""  # Leave empty for AWS S3
        s3ForcePathStyle: false
      gstorage:
        credentialsPath: "/path/to/gcp-credentials.json"
  
  visibility:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"

States:

enabled - Archival active, namespaces can enable
disabled - Archival unavailable
paused - Temporarily suspended

Options:

enableRead - Allow reading from archive

Filestore Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"  # File permissions (octal)
        dirMode: "0766"   # Directory permissions (octal)

URI Format

file:///path/to/archive/directory
file:///mnt/nfs/temporal/archive

Best Practices

Network Storage: Use NFS or similar for multi-node access
Permissions: Ensure all Temporal workers can read/write
Quotas: Monitor disk usage, implement rotation
Backup: Regular backups of archive directory

Example Setup

# Create archive directory
mkdir -p /mnt/temporal-archive/history
mkdir -p /mnt/temporal-archive/visibility

# Set permissions
chmod 0766 /mnt/temporal-archive/history
chmod 0766 /mnt/temporal-archive/visibility

# Set ownership
chown -R temporal:temporal /mnt/temporal-archive

S3 Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"
        endpoint: ""  # Custom endpoint for S3-compatible (e.g., MinIO)
        s3ForcePathStyle: false  # Set true for MinIO

URI Format

s3://bucket-name/path/prefix
s3://temporal-archive-prod/history

Authentication

Use IAM roles (recommended) or credentials:

IAM Role (AWS)
Credentials
MinIO

# No explicit credentials needed
# Attach IAM role to EC2 instances/EKS pods

IAM Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::temporal-archive-prod/*",
        "arn:aws:s3:::temporal-archive-prod"
      ]
    }
  ]
}

# Set environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_SESSION_TOKEN="your-session-token"  # Optional

archival:
  history:
    provider:
      s3store:
        region: "us-east-1"  # Required but ignored by MinIO
        endpoint: "http://minio.example.com:9000"
        s3ForcePathStyle: true

Credentials:

export AWS_ACCESS_KEY_ID="minio-access-key"
export AWS_SECRET_ACCESS_KEY="minio-secret-key"

S3 Bucket Configuration

# Create bucket
aws s3 mb s3://temporal-archive-prod

# Enable versioning (recommended)
aws s3api put-bucket-versioning \
  --bucket temporal-archive-prod \
  --versioning-configuration Status=Enabled

# Set lifecycle policy
aws s3api put-bucket-lifecycle-configuration \
  --bucket temporal-archive-prod \
  --lifecycle-configuration file://lifecycle.json

Lifecycle policy (lifecycle.json):

{
  "Rules": [
    {
      "Id": "TransitionToIA",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 365,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Google Cloud Storage Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      gstorage:
        credentialsPath: "/path/to/service-account-key.json"

URI Format

gs://bucket-name/path/prefix
gs://temporal-archive-prod/history

Authentication

Service Account
Workload Identity (GKE)

archival:
  history:
    provider:
      gstorage:
        credentialsPath: "/etc/temporal/gcp-credentials.json"

Create service account:

# Create service account
gcloud iam service-accounts create temporal-archiver \
  --display-name="Temporal Archiver"

# Grant storage permissions
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:temporal-archiver@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

# Create key
gcloud iam service-accounts keys create gcp-credentials.json \
  --iam-account=temporal-archiver@PROJECT_ID.iam.gserviceaccount.com

# Use Workload Identity instead of key file
# Link Kubernetes service account to GCP service account

gcloud iam service-accounts add-iam-policy-binding \
  temporal-archiver@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[temporal/temporal-worker]"

GCS Bucket Configuration

# Create bucket
gsutil mb -l us-east1 gs://temporal-archive-prod

# Set lifecycle policy
gsutil lifecycle set lifecycle.json gs://temporal-archive-prod

Lifecycle policy (lifecycle.json):

{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
        "condition": {"age": 90}
      },
      {
        "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
        "condition": {"age": 365}
      }
    ]
  }
}

Namespace Configuration

Configure archival per namespace:

# Create namespace with archival
tctl namespace register production \
  --retention 7 \
  --history-archival-state enabled \
  --history-uri "s3://temporal-archive-prod/history/production" \
  --visibility-archival-state enabled \
  --visibility-uri "s3://temporal-archive-prod/visibility/production"

Or update existing:

tctl namespace update production \
  --history-archival-state enabled \
  --history-uri "s3://temporal-archive-prod/history/production"

Namespace States:

enabled - Archive workflows in this namespace
disabled - Do not archive

Default Namespace Archival

Set defaults for new namespaces:

namespaceDefaults:
  archival:
    history:
      state: "disabled"  # disabled, enabled
      URI: "s3://temporal-archive-prod/history"
    visibility:
      state: "disabled"
      URI: "s3://temporal-archive-prod/visibility"

Reading Archived Data

History

Retrieve archived workflow history:

tctl workflow show \
  --workflow-id my-workflow-id \
  --run-id run-id \
  --namespace production

Temporal automatically checks archive if not in primary store.

Visibility

Query archived workflows:

tctl workflow list \
  --namespace production \
  --query 'WorkflowType="MyWorkflow" AND CloseTime > "2024-01-01"'

Note: Visibility queries against archive may be slower than active store.

Archival Process

Archival occurs through system workflows:

Workflow Completes - Workflow reaches final state
Retention Period - Waits for namespace retention period
Archival Task - System worker picks up archival task
Upload History - History events uploaded to archive URI
Upload Visibility - Visibility record uploaded
Delete from DB - Primary store data deleted

Archival Workers

System workers handle archival:

services:
  worker:
    rpc:
      grpcPort: 7239
      membershipPort: 6939

Scale workers based on archival throughput:

# Monitor archival queue size
histogram_quantile(0.99, rate(ArchivalQueueProcessor_latency_bucket[5m]))

Monitoring Archival

Metrics

HistoryArchiver       # History archival operations
VisibilityArchiver    # Visibility archival operations
ArchiverClient        # Archival client operations
ArchivalQueueProcessor # Archival task processing

Each emits:

Request count
Error count
Upload size
Latency

Alerts

Recommended alerts:

Archival Failures

rate(HistoryArchiver_errors[5m]) > 0.01

High Archival Latency

histogram_quantile(0.99, rate(HistoryArchiver_latency_bucket[5m])) > 10000

Queue Backlog

ArchivalQueueProcessor_pending_tasks > 10000

Archival Best Practices

1. Separate Archive URIs per Namespace

# Good
s3://temporal-archive/history/production
s3://temporal-archive/history/staging

# Avoid
s3://temporal-archive/history  # All namespaces

2. Configure Retention Appropriately

# Short retention, rely on archive
tctl namespace register production --retention 7

# Longer retention for critical workflows
tctl namespace register critical --retention 30

3. Test Archive Reads Regularly

# Verify archived data is accessible
tctl workflow show \
  --workflow-id archived-workflow \
  --namespace production

4. Monitor Storage Costs

Use lifecycle policies to transition to cheaper storage
Archive only necessary namespaces
Set appropriate retention periods

5. Implement Archive Backup

Backup archive storage separately:

# S3 cross-region replication
aws s3api put-bucket-replication \
  --bucket temporal-archive-prod \
  --replication-configuration file://replication.json

Troubleshooting

Archival Failures

Symptoms:

HistoryArchiver_errors metrics increasing
Workflows not archived after retention

Common Causes:

Permissions - Missing S3/GCS write permissions
URI Invalid - Incorrect bucket name or path
Network - Cannot reach storage endpoint
Quota - Storage quota exceeded

Solutions:

# Check worker logs
kubectl logs -l app=temporal-worker

# Verify permissions
aws s3 ls s3://temporal-archive-prod/
gsutil ls gs://temporal-archive-prod/

# Test write access
echo "test" | aws s3 cp - s3://temporal-archive-prod/test.txt

Cannot Read Archived Data

Symptoms:

workflow not found when querying old workflows
Archive reads fail

Solutions:

Verify enableRead: true in global config
Check archive URI matches namespace config
Verify read permissions on storage
Check worker can access storage endpoint

High Archival Latency

Symptoms:

Slow archival task processing
Large queue backlog

Solutions:

Scale worker service horizontally
Increase worker task processing concurrency
Use faster storage tier
Optimize network path to storage

Advanced Configuration

Custom Archiver Implementation

Implement custom archiver for proprietary storage:

type HistoryArchiver interface {
    Archive(context.Context, URI, *ArchiveHistoryRequest, ...ArchiveOption) error
    Get(context.Context, URI, *GetHistoryRequest) (*GetHistoryResponse, error)
    ValidateURI(URI) error
}

type VisibilityArchiver interface {
    Archive(context.Context, URI, *ArchiveVisibilityRequest, ...ArchiveOption) error
    Query(context.Context, URI, *QueryVisibilityRequest) (*QueryVisibilityResponse, error)
    ValidateURI(URI) error
}

import "go.temporal.io/server/common/archiver/provider"

func init() {
    provider.RegisterHistoryArchiver("custom", NewCustomHistoryArchiver)
    provider.RegisterVisibilityArchiver("custom", NewCustomVisibilityArchiver)
}

Archival Workflow Customization

Configure archival workflow via dynamic config:

worker.archiverConcurrency:
  - value: 50  # Parallel archival operations
    constraints: {}

worker.archiverMaxPendingArchivalTasks:
  - value: 10000
    constraints: {}

Getting Started

Core Concepts

Deployment

Architecture

Operations

Advanced Features

​Overview

​Supported Providers

​Global Configuration

​Filestore Provider

​Configuration

​URI Format

​Best Practices

​Example Setup

​S3 Provider

​Configuration

​URI Format

​Authentication

​S3 Bucket Configuration

​Google Cloud Storage Provider

​Configuration

​URI Format

​Authentication

​GCS Bucket Configuration

​Namespace Configuration

​Default Namespace Archival

​Reading Archived Data

​History

​Visibility

​Archival Process

​Archival Workers

​Monitoring Archival

​Metrics

​Alerts

​Archival Best Practices

​1. Separate Archive URIs per Namespace

​2. Configure Retention Appropriately

​3. Test Archive Reads Regularly

​4. Monitor Storage Costs

​5. Implement Archive Backup

​Troubleshooting

​Archival Failures

​Cannot Read Archived Data

​High Archival Latency

​Advanced Configuration

​Custom Archiver Implementation

​Archival Workflow Customization

​See Also

Build docs developers (and LLMs) love

Overview

Supported Providers

Global Configuration

Filestore Provider

Configuration

URI Format

Best Practices

Example Setup

S3 Provider

Configuration

URI Format

Authentication

S3 Bucket Configuration

Google Cloud Storage Provider

Configuration

URI Format

Authentication

GCS Bucket Configuration

Namespace Configuration

Default Namespace Archival

Reading Archived Data

History

Visibility

Archival Process

Archival Workers

Monitoring Archival

Metrics

Alerts

Archival Best Practices

1. Separate Archive URIs per Namespace

2. Configure Retention Appropriately

3. Test Archive Reads Regularly

4. Monitor Storage Costs

5. Implement Archive Backup

Troubleshooting

Archival Failures

Cannot Read Archived Data

High Archival Latency

Advanced Configuration

Custom Archiver Implementation

Archival Workflow Customization

See Also