Skip to main content
Hatchet ships built-in OpenTelemetry instrumentation for both Python and Go. When enabled, every task run, workflow trigger, event push, and durable wait operation is wrapped in an OTel span. Spans are automatically linked to one another through W3C traceparent propagation, so a trigger on your web server and the tasks it enqueues appear as a single distributed trace in your observability tool. By default, Hatchet sends traces to its own built-in OTLP collector (visible in the dashboard). You can also send traces to any external collector — Jaeger, Datadog, Honeycomb, Grafana Tempo, and so on — by providing your own TracerProvider.

​
Python

​
Installation

The OTel integration is an optional extra. Install it alongside the core SDK: 
pip install "hatchet-sdk[otel]"
The [otel] extra is required. Importing HatchetInstrumentor without it raises ModuleNotFoundError with a message explaining which packages are missing.

​
Basic setup

Call HatchetInstrumentor().instrument() before creating the Hatchet client and registering any tasks. The instrumentor patches internal SDK methods automatically — no changes to your task functions are needed.
worker.py
from hatchet_sdk import Hatchet, Context, EmptyModel
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor

# Instrument first — before creating the Hatchet client
HatchetInstrumentor().instrument()

hatchet = Hatchet()
With no arguments, HatchetInstrumentor creates an SDK TracerProvider, registers a BatchSpanProcessor that exports to the Hatchet engine’s OTLP endpoint (using the same connection settings as the client), and sets the provider as the global OTel provider.

​
Sending traces to an external collector

Pass your own TracerProvider to forward spans to an additional destination. Traces will be sent to both Hatchet and your external collector:
worker.py
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="your-collector:4317"))
)

HatchetInstrumentor(
    tracer_provider=provider,  # also sends to Hatchet by default
).instrument()
To disable the Hatchet collector entirely and only send to your own backend: 
HatchetInstrumentor(
    tracer_provider=provider,
    enable_hatchet_otel_collector=False,
).instrument()

​
HatchetInstrumentor parameters

ParameterTypeDefaultDescription
tracer_providerTracerProviderNoneCustom provider. If omitted, an SDK provider is created or the existing global one is reused.
meter_providerMeterProviderNoneCustom meter provider. Defaults to a no-op provider.
configClientConfigNoneClient configuration used to derive the Hatchet OTLP endpoint and token.
enable_hatchet_otel_collectorboolTrueWhether to export spans to the Hatchet engine’s built-in OTLP collector.
schedule_delay_millisintOTel defaultDelay between exports in the BatchSpanProcessor.
max_export_batch_sizeintOTel defaultMaximum batch size for the BatchSpanProcessor.
max_queue_sizeintOTel defaultMaximum queue size for the BatchSpanProcessor.

​
Custom child spans

Once instrumented, you can add child spans inside your task functions using the standard OTel API. These spans automatically inherit the Hatchet task span as their parent.
worker.py
import time
from opentelemetry.trace import get_tracer
from hatchet_sdk import Context, EmptyModel, Hatchet
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor

HatchetInstrumentor().instrument()
hatchet = Hatchet()

otel_workflow = hatchet.workflow(name="OTelDataPipeline")

@otel_workflow.task()
def fetch_data(input: EmptyModel, ctx: Context) -> dict:
    tracer = get_tracer(__name__)

    with tracer.start_as_current_span(
        "http.request",
        attributes={"http.method": "GET", "http.url": "https://api.example.com/data"},
    ) as span:
        time.sleep(0.05)
        span.set_attribute("http.status_code", 200)

    with tracer.start_as_current_span("json.parse") as span:
        time.sleep(0.01)
        span.set_attribute("json.record_count", 42)

    return {"records_fetched": 42}

​
Propagating trace context through triggers

When you trigger a workflow from instrumented code, the HatchetInstrumentor automatically injects the W3C traceparent header into the run’s additional_metadata. The worker-side instrumentor extracts it and makes the step run span a child of the triggering span.
trigger.py
from opentelemetry.trace import get_tracer
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
from examples.opentelemetry_instrumentation.worker import otel_workflow

HatchetInstrumentor().instrument()
tracer = get_tracer(__name__)

def main() -> None:
    # The run_workflow call is auto-traced with a "hatchet.run_workflow" span.
    # The traceparent is automatically injected into additional_metadata,
    # so the worker-side spans become children of this trigger span.
    with tracer.start_as_current_span("trigger_otel_data_pipeline"):
        result = otel_workflow.run()
        print(f"Workflow result: {result}")

​
Go

​
Installation

Import the opentelemetry sub-package from the Hatchet Go SDK: 
go get github.com/hatchet-dev/hatchet/sdks/go/opentelemetry

​
Basic setup

Create an Instrumentor and register its middleware on the worker:
worker.go
package main

import (
    "context"
    "log"

    "github.com/hatchet-dev/hatchet/pkg/cmdutils"
    hatchet "github.com/hatchet-dev/hatchet/sdks/go"
    hatchetotel "github.com/hatchet-dev/hatchet/sdks/go/opentelemetry"
)

func main() {
    client, err := hatchet.NewClient()
    if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }

    instrumentor, err := hatchetotel.NewInstrumentor()
    if err != nil {
        log.Fatalf("failed to create instrumentor: %v", err)
    }
    defer instrumentor.Shutdown(context.Background())

    worker, err := client.NewWorker(
        "my-worker",
        hatchet.WithWorkflows(myWorkflow),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Register the OTel middleware
    worker.Use(instrumentor.Middleware())

    interruptCtx, cancel := cmdutils.NewInterruptContext()
    defer cancel()

    if err := worker.StartBlocking(interruptCtx); err != nil {
        log.Fatalf("failed to start worker: %v", err)
    }
}
Connection settings (OTLP endpoint, bearer token, TLS) are read automatically from the same environment variables used by the Hatchet client (HATCHET_CLIENT_HOST_PORT, HATCHET_CLIENT_TOKEN, HATCHET_CLIENT_TLS_STRATEGY).

​
Options

OptionDescription
WithTracerProvider(tp)Use a custom SDK TracerProvider.
DisableHatchetCollector()Do not forward spans to the Hatchet engine.
WithBatchSpanProcessorOptions(opts...)Configure the BatchSpanProcessor used for the Hatchet exporter.
// Use your own TracerProvider and disable the Hatchet collector
instrumentor, err := hatchetotel.NewInstrumentor(
    hatchetotel.WithTracerProvider(myProvider),
    hatchetotel.DisableHatchetCollector(),
)

​
What gets traced

The instrumentor wraps the following operations in OTel spans:
OperationSpan nameKind
Task run starthatchet.start_step_runCONSUMER
Task run cancelhatchet.cancel_step_runCONSUMER
Trigger a workflowhatchet.run_workflowPRODUCER
Trigger multiple workflowshatchet.run_workflowsPRODUCER
Schedule a workflowhatchet.schedule_workflowPRODUCER
Push an eventhatchet.push_eventPRODUCER
Bulk push eventshatchet.bulk_push_eventPRODUCER
Durable waithatchet.durable.wait_forINTERNAL
Every span carries hatchet.* attributes so you can filter by task, workflow, worker, or tenant in your tracing backend:
AttributeDescription
hatchet.workflow_nameName of the workflow
hatchet.step_run_idID of the specific task run
hatchet.workflow_run_idID of the parent workflow run
hatchet.worker_idID of the worker that handled the task
hatchet.tenant_idTenant the run belongs to
hatchet.retry_countNumber of retries at the time of execution
hatchet.payloadInput payload (Python: may be excluded via excluded_attributes)
The HatchetAttributeSpanProcessor (both Python and Go) injects hatchet.* attributes into every child span created within a task run context, so spans you create with get_tracer(__name__) are also queryable by task ID.

​
OTel attributes reference (Python)

The OTelAttribute enum in hatchet_sdk.utils.opentelemetry lists all attribute keys. You can pass values from this enum to exclude specific attributes from spans via the excluded_attributes config option. 
from hatchet_sdk.utils.opentelemetry import OTelAttribute

# Example: exclude the raw payload from all spans
HatchetInstrumentor(
    config=ClientConfig(
        otel=OTelConfig(
            excluded_attributes=[OTelAttribute.ACTION_PAYLOAD]
        )
    )
).instrument()
Selected attributes:
OTelAttribute memberSpan attribute keyDescription
WORKFLOW_NAMEhatchet.workflow_nameWorkflow name
STEP_RUN_IDhatchet.step_run_idTask run ID
WORKFLOW_RUN_IDhatchet.workflow_run_idWorkflow run ID
WORKER_IDhatchet.worker_idWorker ID
TENANT_IDhatchet.tenant_idTenant ID
RETRY_COUNThatchet.retry_countRetry count
ACTION_PAYLOADhatchet.payloadInput payload
EVENT_KEYhatchet.event_keyEvent key (push_event spans)
SIGNAL_KEYhatchet.signal_keySignal key (durable wait spans)

​
Next steps

Dashboard

View traces collected by the Hatchet engine in the dashboard.

Task logs

Correlate OTel traces with task log output.

Build docs developers (and LLMs) love

Get started for freeTalk to us