Exporters - KloudMate Agent

Exporters send processed telemetry data to one or more backends or destinations. The KloudMate Agent includes exporters for sending data to KloudMate and other OTLP-compatible platforms.

OTLP HTTP Exporter

The OTLP HTTP exporter sends telemetry data using the OpenTelemetry Protocol over HTTP. This is the primary exporter for sending data to KloudMate.

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    headers:
      Authorization: ${env:KM_API_KEY}
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000

Configuration Parameters

Parameter	Description	Default
`endpoint`	HTTP endpoint URL	Required
`headers`	Custom HTTP headers (e.g., for authentication)	`{}`
`compression`	Compression algorithm (`gzip`, `none`)	`gzip`
`timeout`	Request timeout	`30s`
`sending_queue.enabled`	Enable persistent queue	`false`
`sending_queue.num_consumers`	Number of parallel consumers	`10`
`sending_queue.queue_size`	Maximum queue size	`1000`
`retry_on_failure.enabled`	Enable retry logic	`true`
`retry_on_failure.initial_interval`	Initial retry delay	`5s`
`retry_on_failure.max_interval`	Maximum retry delay	`30s`
`retry_on_failure.max_elapsed_time`	Maximum total retry time	`5m`

Example with All Options

exporters:
  otlphttp:
    endpoint: https://ingest.kloudmate.com/v1/metrics
    headers:
      Authorization: Bearer ${env:KM_API_KEY}
      X-Custom-Header: custom-value
    compression: gzip
    timeout: 30s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      storage: file_storage
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m

Use Cases

KloudMate Platform: Primary method for sending data to KloudMate
HTTP Proxies: Works through HTTP proxies unlike gRPC
Firewall Compatibility: HTTP/HTTPS typically has better firewall compatibility
Load Balancers: Easier to load balance than gRPC connections

Documentation: OTLP HTTP Exporter

OTLP Exporter (gRPC)

The OTLP gRPC exporter sends telemetry data using gRPC protocol.

exporters:
  otlp:
    endpoint: collector.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca.pem
      cert_file: /etc/ssl/certs/client.pem
      key_file: /etc/ssl/private/client-key.pem
    headers:
      api-key: ${env:API_KEY}

Configuration Parameters

Parameter	Description	Default
`endpoint`	gRPC endpoint (host:port)	Required
`tls.insecure`	Skip TLS verification	`false`
`tls.ca_file`	CA certificate file	-
`tls.cert_file`	Client certificate file	-
`tls.key_file`	Client key file	-
`compression`	Compression (`gzip`, `none`)	`gzip`
`keepalive.time`	Keepalive ping interval	`30s`
`keepalive.timeout`	Keepalive timeout	`10s`

When to Use gRPC vs HTTP

Use OTLP HTTP When

Sending to KloudMate
Behind HTTP proxies
Firewall restrictions
Simpler deployment

Use OTLP gRPC When

Maximum performance needed
Low latency requirements
High throughput scenarios
Direct collector-to-collector

Documentation: OTLP Exporter

Debug Exporter

The debug exporter outputs telemetry data to the console, useful for development and troubleshooting.

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

Verbosity Levels

Level	Description	Output
`basic`	Minimal output	Data type counts only
`normal`	Standard output	Summary information
`detailed`	Full output	Complete telemetry data

Sampling Parameters

sampling_initial: Number of initial items to log
sampling_thereafter: Log every Nth item after initial sampling

Example Output

2024-03-06T10:15:30.123Z info MetricsExporter {"#metrics": 42}
2024-03-06T10:15:30.124Z info LogsExporter {"#logs": 128}

Use Cases:

Testing receiver configurations
Verifying processor transformations
Debugging pipeline issues
Development and local testing

Do not use the debug exporter in production as it can produce large amounts of console output and impact performance.

Documentation: Debug Exporter

Nop Exporter

The no-operation exporter silently drops all data without processing or outputting it.

exporters:
  nop:

Use Cases:

Testing receiver and processor configurations without sending data
Temporary disabling of data export during troubleshooting
Performance testing of receiver/processor throughput
Development scenarios

Documentation: Nop Exporter

Multiple Exporters

You can configure multiple exporters in a single pipeline to send data to multiple destinations.

exporters:
  otlphttp/kloudmate:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    headers:
      Authorization: ${env:KM_API_KEY}
  
  otlphttp/backup:
    endpoint: ${env:BACKUP_ENDPOINT}
    headers:
      Authorization: ${env:BACKUP_API_KEY}
  
  debug:
    verbosity: basic

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [otlphttp/kloudmate, otlphttp/backup, debug]

Use Cases:

Primary and backup destinations
Multi-tenancy (sending to different platforms)
Compliance (archiving to long-term storage)
Development (debug exporter + production exporter)

Each exporter processes data independently. If one exporter fails, others continue to operate.

Exporter Queue and Retry Configuration

Persistent Queue

Enable persistent queuing to prevent data loss during network outages or backend unavailability.

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      storage: file_storage  # Reference to file_storage extension

extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage
    timeout: 10s

service:
  extensions: [file_storage]

Benefits:

Data persists across collector restarts
Survives temporary network failures
Handles backend downtime gracefully

Retry Configuration

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m

Retry Behavior:

Initial failure triggers retry after initial_interval
Subsequent retries use exponential backoff
Backoff capped at max_interval
Stops retrying after max_elapsed_time

Authentication Methods

API Key Authentication

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    headers:
      Authorization: Bearer ${env:KM_API_KEY}

Basic Authentication

exporters:
  otlphttp:
    endpoint: ${env:ENDPOINT}
    headers:
      Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=  # base64(username:password)

Custom Headers

exporters:
  otlphttp:
    endpoint: ${env:ENDPOINT}
    headers:
      X-API-Key: ${env:API_KEY}
      X-Tenant-ID: ${env:TENANT_ID}
      X-Environment: production

Always use environment variables for sensitive data like API keys. Never hardcode credentials in configuration files.

TLS Configuration

Server Certificate Verification

exporters:
  otlp:
    endpoint: collector.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca.pem

Mutual TLS (mTLS)

exporters:
  otlp:
    endpoint: collector.example.com:4317
    tls:
      ca_file: /etc/ssl/certs/ca.pem
      cert_file: /etc/ssl/certs/client.pem
      key_file: /etc/ssl/private/client-key.pem

Skip Verification (Development Only)

exporters:
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true

Only use insecure: true in development environments. Production deployments should always verify TLS certificates.

Performance Tuning

High Throughput Configuration

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    sending_queue:
      enabled: true
      num_consumers: 20        # More parallel senders
      queue_size: 50000        # Larger queue
    timeout: 60s               # Longer timeout
    compression: gzip          # Enable compression

Low Latency Configuration

exporters:
  otlp:
    endpoint: ${env:ENDPOINT}
    sending_queue:
      enabled: false           # No queuing
    compression: none          # Skip compression
    keepalive:
      time: 10s                # Frequent keepalives
      timeout: 5s

Resource-Constrained Environments

exporters:
  otlphttp:
    endpoint: ${env:KM_COLLECTOR_ENDPOINT}
    sending_queue:
      enabled: true
      num_consumers: 5         # Fewer consumers
      queue_size: 5000         # Smaller queue
    timeout: 30s

Monitoring Exporter Health

Exporters expose metrics that help monitor their health and performance:

Metric	Description
`otelcol_exporter_sent_spans`	Number of spans successfully sent
`otelcol_exporter_sent_metric_points`	Number of metric points sent
`otelcol_exporter_sent_log_records`	Number of log records sent
`otelcol_exporter_send_failed_spans`	Number of spans that failed to send
`otelcol_exporter_send_failed_metric_points`	Number of metric points that failed
`otelcol_exporter_queue_size`	Current queue size
`otelcol_exporter_queue_capacity`	Maximum queue capacity

Enable collector self-monitoring to track exporter metrics and detect issues early.

Best Practices

Use environment variables for endpoints and credentials

Keep configuration portable and secure by externalizing sensitive values

Enable persistent queues for production

Prevent data loss during network issues or collector restarts

Configure appropriate timeouts

Balance between allowing time for slow networks and detecting failures quickly

Enable compression

Reduce network bandwidth usage, especially for high-volume deployments

Monitor exporter metrics

Track queue sizes, send failures, and latency to detect issues proactively

Test failover scenarios

Verify that retry logic and persistent queues work as expected

Troubleshooting

Exporter Not Sending Data

Check endpoint connectivity: curl ${KM_COLLECTOR_ENDPOINT}
Verify authentication headers are correct
Check collector logs for error messages
Enable debug exporter to verify data reaches the pipeline

High Queue Size

Backend may be slow or unavailable
Increase num_consumers for more parallelism
Check network latency to backend
Verify backend can handle the data volume

Frequent Timeouts

Increase timeout value
Check network latency and stability
Reduce batch size in batch processor
Verify backend health and capacity

Next Steps

Configure Extensions

Add health checks and storage capabilities

Pipeline Configuration

Learn how to build complete pipelines

OpenTelemetry Components

​OTLP HTTP Exporter

​Configuration Parameters

​Example with All Options

​Use Cases

​OTLP Exporter (gRPC)

​Configuration Parameters

​When to Use gRPC vs HTTP

Use OTLP HTTP When

Use OTLP gRPC When

​Debug Exporter

​Verbosity Levels

​Sampling Parameters

​Example Output

​Nop Exporter

​Multiple Exporters

​Exporter Queue and Retry Configuration

​Persistent Queue

​Retry Configuration

​Authentication Methods

​API Key Authentication

​Basic Authentication

​Custom Headers

​TLS Configuration

​Server Certificate Verification

​Mutual TLS (mTLS)

​Skip Verification (Development Only)

​Performance Tuning

​High Throughput Configuration

​Low Latency Configuration

​Resource-Constrained Environments

​Monitoring Exporter Health

​Best Practices

​Troubleshooting

​Exporter Not Sending Data

​High Queue Size

​Frequent Timeouts

​Next Steps

Configure Extensions

Pipeline Configuration

Build docs developers (and LLMs) love

OTLP HTTP Exporter

Configuration Parameters

Example with All Options

Use Cases

OTLP Exporter (gRPC)

Configuration Parameters

When to Use gRPC vs HTTP

Debug Exporter

Verbosity Levels

Sampling Parameters

Example Output

Nop Exporter

Multiple Exporters

Exporter Queue and Retry Configuration

Persistent Queue

Retry Configuration

Authentication Methods

API Key Authentication

Basic Authentication

Custom Headers

TLS Configuration

Server Certificate Verification

Mutual TLS (mTLS)

Skip Verification (Development Only)

Performance Tuning

High Throughput Configuration

Low Latency Configuration

Resource-Constrained Environments

Monitoring Exporter Health

Best Practices

Troubleshooting

Exporter Not Sending Data

High Queue Size

Frequent Timeouts

Next Steps