OpenTelemetry Integration

Gate includes built-in support for OpenTelemetry, providing comprehensive observability through metrics, distributed tracing, and structured logging. OpenTelemetry is a vendor-agnostic observability framework that allows you to instrument your application and export telemetry data to various backends.

Overview

OpenTelemetry is an observability framework designed to facilitate the generation, export, and collection of telemetry data such as traces, metrics, and logs. Gate leverages the otel-config-go library, which provides a straightforward method to configure observability through environment variables.

Gate automatically initializes OpenTelemetry with sensible defaults. You only need to configure the endpoints and credentials for your observability backend.

Architecture

The typical Gate observability setup follows this architecture:

Gate → OpenTelemetry Collector → Observability Backends
                                  ├── Prometheus (Metrics)
                                  ├── Tempo/Jaeger (Traces)
                                  └── Loki (Logs)

Gate emits telemetry data using the OpenTelemetry Protocol (OTLP), which can be sent to:

OpenTelemetry Collector (recommended) - Processes and routes telemetry
Direct backends - Services like Grafana Cloud, Honeycomb, or Datadog
Self-hosted solutions - Prometheus, Jaeger, Tempo, etc.

Configuration

Gate’s OpenTelemetry implementation is configured entirely through environment variables, making it easy to integrate with container orchestration platforms and cloud environments.

Core Settings

Environment Variable	Default	Description
`OTEL_SERVICE_NAME`	`gate`	Name of your service
`OTEL_SERVICE_VERSION`	-	Version of your service
`OTEL_EXPORTER_OTLP_ENDPOINT`	`localhost:4317`	Endpoint for OTLP export
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	Protocol for OTLP (`grpc` or `http/protobuf`)
`OTEL_METRICS_ENABLED`	`false`	Enable metrics collection
`OTEL_TRACES_ENABLED`	`false`	Enable trace collection
`OTEL_LOG_LEVEL`	`info`	OpenTelemetry SDK logging level
`OTEL_PROPAGATORS`	`tracecontext,baggage`	Configured trace propagators

Exporter Configuration

Environment Variable	Default	Description
`OTEL_EXPORTER_OTLP_HEADERS`	`{}`	Global headers for OTLP exporter
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	`localhost:4317`	Endpoint for trace export
`OTEL_EXPORTER_OTLP_TRACES_HEADERS`	`{}`	Headers specific to trace exporter
`OTEL_EXPORTER_OTLP_TRACES_INSECURE`	`false`	Allow insecure trace connections
`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`	`localhost:4317`	Endpoint for metrics export
`OTEL_EXPORTER_OTLP_METRICS_HEADERS`	`{}`	Headers specific to metrics exporter
`OTEL_EXPORTER_OTLP_METRICS_INSECURE`	`false`	Allow insecure metrics connections
`OTEL_EXPORTER_OTLP_METRICS_PERIOD`	`30s`	Metrics reporting interval

Resource Attributes

Use OTEL_RESOURCE_ATTRIBUTES to add additional context to your telemetry:

# Single attribute
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production"

# Multiple attributes
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,cloud.region=us-east-1,service.instance.id=gate-1"

Quick Start Examples

Local OpenTelemetry Collector

export OTEL_SERVICE_NAME="gate-proxy"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_METRICS_ENABLED="true"
export OTEL_TRACES_ENABLED="true"

gate

Grafana Cloud

export OTEL_SERVICE_NAME="gate-proxy-prod"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-us-central-0.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64_credentials>"
export OTEL_METRICS_ENABLED="true"
export OTEL_TRACES_ENABLED="true"

gate

Honeycomb

export OTEL_SERVICE_NAME="gate-proxy"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=<your-api-key>"
export OTEL_METRICS_ENABLED="true"
export OTEL_TRACES_ENABLED="true"

gate

Implementation Details

Gate’s OpenTelemetry integration is implemented in pkg/internal/otelutil/otel.go:15:

func Init(ctx context.Context) (clean func(), err error) {
    serviceName := os.Getenv("OTEL_SERVICE_NAME")
    if serviceName == "" {
        serviceName = "gate"
    }
    
    otelShutdown, err := otelconfig.ConfigureOpenTelemetry(
        otelconfig.WithServiceName(serviceName),
        otelconfig.WithServiceVersion(version.String()),
        otelconfig.WithMetricsEnabled(os.Getenv("OTEL_METRICS_ENABLED") == "true"),
        otelconfig.WithTracesEnabled(os.Getenv("OTEL_TRACES_ENABLED") == "true"),
    )
    // ...
}

The SDK automatically:

Initializes trace and metric providers
Configures OTLP exporters (gRPC or HTTP)
Sets up context propagation
Handles graceful shutdown

Best Practices

1. Use Meaningful Service Names

# Good - Identifies the specific instance and purpose
export OTEL_SERVICE_NAME="gate-proxy-lobby-us-east"

# Bad - Too generic
export OTEL_SERVICE_NAME="gate"

2. Track Service Versions

# Semantic versioning
export OTEL_SERVICE_VERSION="v1.2.3"

# Git commit hash
export OTEL_SERVICE_VERSION="$(git rev-parse --short HEAD)"

# Build number
export OTEL_SERVICE_VERSION="build-${BUILD_NUMBER}"

3. Add Deployment Context

export OTEL_RESOURCE_ATTRIBUTES="\
deployment.environment=production,\
cloud.provider=aws,\
cloud.region=us-east-1,\
service.namespace=minecraft,\
service.instance.id=${HOSTNAME}"

4. Secure Production Endpoints

# Always use HTTPS in production
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel-collector.example.com:4317"

# Use authentication headers
export OTEL_EXPORTER_OTLP_HEADERS="api-key=${SECRET_API_KEY}"

5. Use OpenTelemetry Collector

For production deployments, route telemetry through an OpenTelemetry Collector for:

Batching - Reduces network overhead
Filtering - Removes unnecessary data
Enrichment - Adds metadata and context
Retry logic - Handles transient failures
Multi-backend support - Send to multiple destinations

Observability Solutions

Gate works with any OpenTelemetry-compatible backend:

Cloud Platforms

Grafana Cloud - Fully managed with Prometheus, Tempo, and Loki
Honeycomb - Observability for debugging complex systems
Datadog - Full-stack monitoring and analytics
New Relic - APM and observability platform
Azure Monitor - Microsoft cloud monitoring
AWS X-Ray - AWS distributed tracing
Google Cloud Observability - GCP monitoring

Self-Hosted

Metrics: Prometheus, Grafana Mimir
Tracing: Jaeger, Grafana Tempo
Logs: Loki
Visualization: Grafana

Troubleshooting

No Telemetry Data

Verify metrics/traces are enabled:

echo $OTEL_METRICS_ENABLED
echo $OTEL_TRACES_ENABLED

Check the endpoint is reachable:

curl -v http://localhost:4318/v1/traces

Review Gate logs for OpenTelemetry errors

Connection Refused

Ensure the collector/backend is running
Verify the endpoint URL and port
Check firewall rules
For Docker, ensure containers are on the same network

Authentication Errors

Verify API keys and tokens are correct
Check header format: key=value pairs
Ensure credentials are base64 encoded if required

Next Steps

Metrics

Learn about available metrics and collection

Tracing

Configure distributed tracing for request flows

Logging

Set up structured logging and log levels

Operations

Docker

Kubernetes

Observability

Advanced

OpenTelemetry Integration

OpenTelemetry Integration

Overview

Architecture

Configuration

Core Settings

Exporter Configuration

Resource Attributes

Quick Start Examples

Local OpenTelemetry Collector

Grafana Cloud

Honeycomb

Implementation Details

Best Practices

1. Use Meaningful Service Names

2. Track Service Versions

3. Add Deployment Context

4. Secure Production Endpoints

5. Use OpenTelemetry Collector

Observability Solutions

Cloud Platforms

Self-Hosted

Troubleshooting

No Telemetry Data

Connection Refused

Authentication Errors

Next Steps

Metrics

Tracing

Logging

Further Reading

Build docs developers (and LLMs) love

Operations

Docker

Kubernetes

Observability

Advanced

​OpenTelemetry Integration

​Overview

​Architecture

​Configuration

​Core Settings

​Exporter Configuration

​Resource Attributes

​Quick Start Examples

​Local OpenTelemetry Collector

​Grafana Cloud

​Honeycomb

​Implementation Details

​Best Practices

​1. Use Meaningful Service Names

​2. Track Service Versions

​3. Add Deployment Context

​4. Secure Production Endpoints

​5. Use OpenTelemetry Collector

​Observability Solutions

​Cloud Platforms

​Self-Hosted

​Troubleshooting

​No Telemetry Data

​Connection Refused

​Authentication Errors

​Next Steps

Metrics

Tracing

Logging

​Further Reading

Build docs developers (and LLMs) love

OpenTelemetry Integration

Overview

Architecture

Configuration

Core Settings

Exporter Configuration

Resource Attributes

Quick Start Examples

Local OpenTelemetry Collector

Grafana Cloud

Honeycomb

Implementation Details

Best Practices

1. Use Meaningful Service Names

2. Track Service Versions

3. Add Deployment Context

4. Secure Production Endpoints

5. Use OpenTelemetry Collector

Observability Solutions

Cloud Platforms

Self-Hosted

Troubleshooting

No Telemetry Data

Connection Refused

Authentication Errors

Next Steps

Further Reading