Tracing Middleware

Overview

The tracing middleware provides OpenTelemetry distributed tracing support for Kratos applications. It automatically creates spans for incoming requests (server) and outgoing requests (client), propagates trace context, and records span attributes.

Installation

go get github.com/go-kratos/kratos/v2/middleware/tracing

Server Middleware

The Server function creates a server-side tracing middleware that instruments incoming requests:

func Server(opts ...Option) middleware.Middleware

Basic Usage

import (
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "github.com/go-kratos/kratos/v2/transport/http"
    "github.com/go-kratos/kratos/v2/transport/grpc"
    
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/resource"
    tracesdk "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

// Initialize tracer provider
func initTracer(serviceName string) (*tracesdk.TracerProvider, error) {
    // Create Jaeger exporter
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }
    
    // Create tracer provider
    tp := tracesdk.NewTracerProvider(
        tracesdk.WithBatcher(exporter),
        tracesdk.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )
    
    // Set global tracer provider
    otel.SetTracerProvider(tp)
    
    return tp, nil
}

func main() {
    // Initialize tracer
    tp, err := initTracer("my-service")
    if err != nil {
        log.Fatal(err)
    }
    defer tp.Shutdown(context.Background())
    
    // Create HTTP server with tracing
    httpSrv := http.NewServer(
        http.Address(":8000"),
        http.Middleware(
            tracing.Server(),
        ),
    )
    
    // Create gRPC server with tracing
    grpcSrv := grpc.NewServer(
        grpc.Address(":9000"),
        grpc.Middleware(
            tracing.Server(),
        ),
    )
    
    app := kratos.New(
        kratos.Server(httpSrv, grpcSrv),
    )
    
    if err := app.Run(); err != nil {
        log.Fatal(err)
    }
}

Client Middleware

The Client function creates a client-side tracing middleware for outgoing requests:

func Client(opts ...Option) middleware.Middleware

Usage Example

import (
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "github.com/go-kratos/kratos/v2/transport/http"
)

// Create HTTP client with tracing
conn, err := http.NewClient(
    context.Background(),
    http.WithEndpoint("127.0.0.1:8000"),
    http.WithMiddleware(
        tracing.Client(),
    ),
)
if err != nil {
    log.Fatal(err)
}

// Create gRPC client with tracing
conn, err := grpc.DialInsecure(
    context.Background(),
    grpc.WithEndpoint("127.0.0.1:9000"),
    grpc.WithMiddleware(
        tracing.Client(),
    ),
)

Configuration Options

The tracing middleware supports several configuration options:

WithTracerProvider

Specify a custom tracer provider:

import (
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "go.opentelemetry.io/otel/trace"
)

// Use custom tracer provider
tracing.Server(
    tracing.WithTracerProvider(customProvider),
)

By default, the middleware uses the global tracer provider set via otel.SetTracerProvider(provider).

WithTracerName

Set a custom tracer name:

tracing.Server(
    tracing.WithTracerName("my-custom-tracer"),
)

WithPropagator

Configure trace context propagation:

import (
    "go.opentelemetry.io/otel/propagation"
)

// Use W3C Trace Context propagation
tracing.Server(
    tracing.WithPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    ),
)

Trace Context in Logs

The tracing middleware provides valuers to include trace IDs in logs:

import (
    "github.com/go-kratos/kratos/v2/log"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

// TraceID returns a traceid valuer
func TraceID() log.Valuer

// SpanID returns a spanid valuer
func SpanID() log.Valuer

Usage Example

import (
    "github.com/go-kratos/kratos/v2/log"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
)

// Create logger with trace context
logger := log.With(
    log.NewStdLogger(os.Stdout),
    "trace.id", tracing.TraceID(),
    "span.id", tracing.SpanID(),
    "service.name", "my-service",
)

// Use in middleware
logging.Server(logger)

This will automatically include trace and span IDs in all log entries:

{"level":"INFO","trace.id":"4bf92f3577b34da6a3ce929d0e0e4736","span.id":"00f067aa0ba902b7","service.name":"my-service","msg":"request completed"}

Complete Example

Here’s a complete example with Jaeger exporter:

package main

import (
    "context"
    "log"
    "os"
    
    "github.com/go-kratos/kratos/v2"
    "github.com/go-kratos/kratos/v2/log"
    "github.com/go-kratos/kratos/v2/middleware/logging"
    "github.com/go-kratos/kratos/v2/middleware/tracing"
    "github.com/go-kratos/kratos/v2/transport/http"
    
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    tracesdk "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func initTracer(serviceName string) (*tracesdk.TracerProvider, error) {
    // Create Jaeger exporter
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }
    
    // Create tracer provider
    tp := tracesdk.NewTracerProvider(
        tracesdk.WithBatcher(exporter),
        tracesdk.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
        tracesdk.WithSampler(tracesdk.AlwaysSample()),
    )
    
    // Set global providers
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    )
    
    return tp, nil
}

func main() {
    // Initialize tracer
    tp, err := initTracer("my-service")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }()
    
    // Create logger with trace context
    logger := log.With(
        log.NewStdLogger(os.Stdout),
        "trace.id", tracing.TraceID(),
        "span.id", tracing.SpanID(),
        "service", "my-service",
    )
    
    // Create HTTP server with tracing and logging
    httpSrv := http.NewServer(
        http.Address(":8000"),
        http.Middleware(
            tracing.Server(),
            logging.Server(logger),
        ),
    )
    
    app := kratos.New(
        kratos.Name("my-service"),
        kratos.Server(httpSrv),
    )
    
    if err := app.Run(); err != nil {
        log.Fatal(err)
    }
}

OpenTelemetry Exporters

Jaeger

go get go.opentelemetry.io/otel/exporters/jaeger

import "go.opentelemetry.io/otel/exporters/jaeger"

exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
    jaeger.WithEndpoint("http://localhost:14268/api/traces"),
))

OTLP (OpenTelemetry Protocol)

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

exporter, err := otlptracegrpc.New(
    context.Background(),
    otlptracegrpc.WithEndpoint("localhost:4317"),
    otlptracegrpc.WithInsecure(),
)

Zipkin

go get go.opentelemetry.io/otel/exporters/zipkin

import "go.opentelemetry.io/otel/exporters/zipkin"

exporter, err := zipkin.New(
    "http://localhost:9411/api/v2/spans",
)

Span Attributes

The tracing middleware automatically records the following span attributes:

rpc.service - Service name
rpc.method - Operation name
rpc.grpc.status_code - gRPC status code (for gRPC transport)
http.status_code - HTTP status code (for HTTP transport)
http.method - HTTP method
http.url - Request URL
http.target - Request target

Best Practices

Use Sampling in Production

Always use sampling in production to reduce overhead and storage costs. Consider using probability-based or rate-limiting samplers.

tracesdk.WithSampler(tracesdk.TraceIDRatioBased(0.1)) // Sample 10% of traces

Include Trace IDs in Logs

Always include trace and span IDs in logs for correlation. Use the TraceID() and SpanID() valuers.

Propagate Context

Always pass context.Context through your application to ensure trace context is properly propagated.

Add Custom Spans

Add custom spans for important operations within your handlers:

import "go.opentelemetry.io/otel"

func (s *service) DoSomething(ctx context.Context) error {
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "DoSomething")
    defer span.End()
    
    // Your code here
    return nil
}

Handle Shutdown Gracefully

Always shutdown the tracer provider gracefully to ensure all spans are exported.

Source Reference

The tracing middleware implementation can be found in:

middleware/tracing/tracing.go:46 - Server middleware
middleware/tracing/tracing.go:63 - Client middleware
middleware/tracing/tracing.go:78 - TraceID valuer
middleware/tracing/tracing.go:88 - SpanID valuer

Next Steps

Metrics

Add Prometheus metrics to your services

Logging

Configure structured logging

Get Started

Core Concepts

Components

Middleware

Guides

CLI Reference

Tracing Middleware

Overview

Installation

Server Middleware

Basic Usage

Client Middleware

Usage Example

Configuration Options

WithTracerProvider

WithTracerName

WithPropagator

Trace Context in Logs

Usage Example

Complete Example

OpenTelemetry Exporters

Jaeger

OTLP (OpenTelemetry Protocol)

Zipkin

Span Attributes

Best Practices

Source Reference

Next Steps

Metrics

Logging

Build docs developers (and LLMs) love

Get Started

Core Concepts

Components

Middleware

Guides

CLI Reference

​Overview

​Installation

​Server Middleware

​Basic Usage

​Client Middleware

​Usage Example

​Configuration Options

​WithTracerProvider

​WithTracerName

​WithPropagator

​Trace Context in Logs

​Usage Example

​Complete Example

​OpenTelemetry Exporters

​Jaeger

​OTLP (OpenTelemetry Protocol)

​Zipkin

​Span Attributes

​Best Practices

​Source Reference

​Next Steps

Metrics

Logging

Build docs developers (and LLMs) love

Overview

Installation

Server Middleware

Basic Usage

Client Middleware

Usage Example

Configuration Options

WithTracerProvider

WithTracerName

WithPropagator

Trace Context in Logs

Usage Example

Complete Example

OpenTelemetry Exporters

Jaeger

OTLP (OpenTelemetry Protocol)

Zipkin

Span Attributes

Best Practices

Source Reference

Next Steps