Log Processors - OpenTelemetry Rust

Overview

Log processors are responsible for receiving log records from loggers and forwarding them to exporters. OpenTelemetry SDK provides two built-in processors:

SimpleLogProcessor: Exports logs immediately (for development/debugging)
BatchLogProcessor: Buffers logs and exports in batches (for production)

+-----+---------------+   +-----------------------+   +-------------------+
|     |               |   |                       |   |                   |
| SDK | Logger.emit() +---> (Simple)LogProcessor  +--->  LogExporter      |
|     |               |   | (Batch)LogProcessor   +--->  (OTLPExporter)   |
+-----+---------------+   +-----------------------+   +-------------------+

LogProcessor Trait

All processors implement the LogProcessor trait:

pub trait LogProcessor: Send + Sync + Debug {
    /// Called when a log record is ready to be processed and exported
    fn emit(&self, record: &mut SdkLogRecord, scope: &InstrumentationScope);

    /// Force flush cached logs to the exporter
    fn force_flush(&self) -> OTelSdkResult;

    /// Shutdown the processor
    fn shutdown_with_timeout(&self, timeout: Duration) -> OTelSdkResult;

    /// Check if logging is enabled for given severity/target/name
    fn event_enabled(&self, level: Severity, target: &str, name: Option<&str>) -> bool {
        true // Default: all events enabled
    }

    /// Set the resource for the log processor
    fn set_resource(&mut self, resource: &Resource) {}
}

SimpleLogProcessor

The SimpleLogProcessor exports log records immediately when they are emitted. It’s synchronous and suitable for debugging and testing, but not recommended for production due to performance overhead.

Characteristics

Synchronous export on each log emission
No batching or buffering
Immediate visibility of logs
Higher overhead per log record
Simpler error handling

Usage

use opentelemetry_sdk::logs::{SdkLoggerProvider, SimpleLogProcessor};
use opentelemetry_stdout::LogExporter;

let exporter = LogExporter::default();
let provider = SdkLoggerProvider::builder()
    .with_simple_exporter(exporter)
    .build();

Or manually:

use opentelemetry_sdk::logs::SimpleLogProcessor;

let exporter = LogExporter::default();
let processor = SimpleLogProcessor::new(exporter);

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

When to Use

Development & Debugging

Use SimpleLogProcessor when:

Developing and debugging locally
Running tests that need immediate log visibility
Prototyping with small log volumes
Using simple exporters like stdout

NOT for Production

Avoid in production because:

Each log blocks the emitting thread
High overhead for high-throughput scenarios
Network latency affects application performance
No batching efficiency

Runtime Compatibility

When using SimpleLogProcessor with different OTLP exporter clients:

Exporter Feature	Runtime Required	Emit From
`grpc-tonic`	Tokio runtime	Any thread
`reqwest-blocking-client`	No runtime needed	Non-tokio threads only
`reqwest-client`	Tokio runtime	Tokio runtime threads

Deadlock Risk: Using async exporters (e.g., with tokio::sleep) in SimpleLogProcessor can cause deadlocks if all runtime worker threads are blocked. Use BatchLogProcessor for async exporters.

BatchLogProcessor

The BatchLogProcessor buffers log records in memory and exports them in batches using a background thread. This is the recommended processor for production environments.

Characteristics

Asynchronous export via background thread
Configurable batching and scheduling
Reduced export overhead
Bounded memory usage
Optimal for high-throughput scenarios

Basic Usage

use opentelemetry_sdk::logs::{BatchLogProcessor, SdkLoggerProvider};
use opentelemetry_stdout::LogExporter;

let exporter = LogExporter::default();
let processor = BatchLogProcessor::builder(exporter).build();

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

Configuration

Configure batch behavior using BatchConfigBuilder:

use opentelemetry_sdk::logs::{BatchLogProcessor, BatchConfigBuilder};
use std::time::Duration;

let exporter = LogExporter::default();

let processor = BatchLogProcessor::builder(exporter)
    .with_batch_config(
        BatchConfigBuilder::default()
            .with_max_queue_size(2048)           // Max buffered logs
            .with_max_export_batch_size(512)     // Max logs per export
            .with_scheduled_delay(Duration::from_secs(5))  // Export interval
            .build(),
    )
    .build();

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

Configuration Parameters

Parameter	Default	Environment Variable	Description
`max_queue_size`	2048	`OTEL_BLRP_MAX_QUEUE_SIZE`	Maximum number of logs buffered in queue
`max_export_batch_size`	512	`OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`	Maximum logs in a single export batch
`scheduled_delay`	1s	`OTEL_BLRP_SCHEDULE_DELAY`	Delay between consecutive exports

max_export_batch_size must be less than or equal to max_queue_size.

Export Triggers

Logs are exported when:

Batch size reached: When max_export_batch_size logs are buffered
Scheduled delay: Every scheduled_delay milliseconds
Force flush: When force_flush() is called
Shutdown: When shutdown() is called

Example: Production Configuration

use opentelemetry_sdk::logs::{BatchLogProcessor, BatchConfigBuilder};
use std::time::Duration;

let processor = BatchLogProcessor::builder(exporter)
    .with_batch_config(
        BatchConfigBuilder::default()
            .with_max_queue_size(4096)        // Higher capacity
            .with_max_export_batch_size(1024) // Larger batches
            .with_scheduled_delay(Duration::from_secs(10))  // Less frequent exports
            .build(),
    )
    .build();

Runtime Compatibility

When using BatchLogProcessor with OTLP exporters:

Exporter Feature	Runtime Required	Notes
`grpc-tonic`	Tokio runtime	LoggerProvider created in tokio runtime
`reqwest-blocking-client`	No runtime needed	Works with regular `main` or `tokio::main`
`reqwest-client`	Not supported	Use `grpc-tonic` or `reqwest-blocking-client`
`hyper`	Not supported	Use `grpc-tonic` or `reqwest-blocking-client`

Force Flush

Explicitly export buffered logs:

// Flush all buffered logs
provider.force_flush().unwrap();

This is useful before:

Checkpointing application state
Deploying new versions
Handling critical errors

Shutdown

Always shutdown before application exit:

// Flush and shutdown
provider.shutdown().unwrap();

Tokio Deadlock: When using tokio’s current_thread runtime, call shutdown() from a separate thread or use spawn_blocking, as it’s a blocking call that can deadlock.

// Safe shutdown with current_thread runtime
tokio::task::spawn_blocking(move || {
    provider.shutdown().unwrap();
}).await.unwrap();

Dropped Logs

If the queue is full, logs are dropped and counted. The count is logged at shutdown:

// If queue is full, logs are dropped
// Increase max_queue_size or export more frequently

Monitor dropped logs and adjust configuration as needed.

Custom Log Processors

You can implement custom processors for specialized behavior:

use opentelemetry::logs::Severity;
use opentelemetry::InstrumentationScope;
use opentelemetry_sdk::error::OTelSdkResult;
use opentelemetry_sdk::logs::{LogProcessor, SdkLogRecord};
use opentelemetry_sdk::Resource;
use std::time::Duration;

#[derive(Debug)]
struct FilteringProcessor<P: LogProcessor> {
    delegate: P,
    min_severity: Severity,
}

impl<P: LogProcessor> LogProcessor for FilteringProcessor<P> {
    fn emit(&self, record: &mut SdkLogRecord, scope: &InstrumentationScope) {
        // Only forward logs at or above min_severity
        if record.severity_number().unwrap_or(Severity::Info) >= self.min_severity {
            self.delegate.emit(record, scope);
        }
    }

    fn force_flush(&self) -> OTelSdkResult {
        self.delegate.force_flush()
    }

    fn shutdown_with_timeout(&self, timeout: Duration) -> OTelSdkResult {
        self.delegate.shutdown_with_timeout(timeout)
    }

    fn event_enabled(&self, level: Severity, target: &str, name: Option<&str>) -> bool {
        level >= self.min_severity && self.delegate.event_enabled(level, target, name)
    }

    fn set_resource(&mut self, resource: &Resource) {
        self.delegate.set_resource(resource);
    }
}

Usage:

use opentelemetry_sdk::logs::SimpleLogProcessor;

let simple = SimpleLogProcessor::new(exporter);
let filtering = FilteringProcessor {
    delegate: simple,
    min_severity: Severity::Warn,
};

let provider = SdkLoggerProvider::builder()
    .with_log_processor(filtering)
    .build();

event_enabled Optimization

The event_enabled() method allows early filtering to skip expensive logging operations:

if !processor.event_enabled(Severity::Debug, "my-module", Some("my-event")) {
    return; // Skip expensive work
}

// Only execute if event is enabled
let expensive_data = compute_expensive_data();
log_record.add_attribute("data", expensive_data);

Processor Comparison

Feature	SimpleLogProcessor	BatchLogProcessor
Export Mode	Synchronous	Asynchronous (background)
Buffering	None	Yes
Overhead	High (per log)	Low (amortized)
Latency	None	Configurable
Memory Usage	Minimal	Bounded by queue size
Production Ready	No	Yes
Best For	Development, debugging	Production
Dropped Logs	Never	If queue full
Background Thread	No	Yes

Best Practices

Use BatchLogProcessor in Production

Always use BatchLogProcessor for production deployments:

Lower overhead
Better performance under load
Configurable batching behavior

Configure Based on Load

Adjust batch configuration based on your log volume:

// High-volume service
.with_max_queue_size(8192)
.with_max_export_batch_size(2048)
.with_scheduled_delay(Duration::from_secs(2))

// Low-volume service
.with_max_queue_size(512)
.with_max_export_batch_size(128)
.with_scheduled_delay(Duration::from_secs(30))

Always Shutdown Cleanly

Call shutdown() before application exit to flush remaining logs:

// Register shutdown hook
let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

// On exit
provider.shutdown().unwrap();

Monitor Dropped Logs

Check for dropped log messages in production and adjust max_queue_size if needed.

Use event_enabled for Performance

Implement event_enabled() in custom processors to skip expensive operations for filtered logs.

Troubleshooting

Logs not exported

Forgot to shutdown: Always call provider.shutdown()
```
provider.shutdown().unwrap();
```
Batch not full: Wait for scheduled_delay or call force_flush()
```
provider.force_flush().unwrap();
```
Queue full: Increase max_queue_size or export more frequently

High latency

If using SimpleLogProcessor in production, switch to BatchLogProcessor:

let processor = BatchLogProcessor::builder(exporter).build();

Memory usage too high

Reduce max_queue_size or increase export frequency:

.with_max_queue_size(1024)  // Smaller buffer
.with_scheduled_delay(Duration::from_secs(1))  // More frequent exports

Logs dropped

Increase queue size or export more frequently:

.with_max_queue_size(8192)  // Larger buffer
.with_scheduled_delay(Duration::from_millis(500))  // More frequent

log Appender

Bridge logs from the log crate

tracing Appender

Bridge logs from the tracing crate

Bridge API

Understanding the core API

Overview

Return to logs overview

Get Started

Core Concepts

Traces

Metrics

Logs

Exporters

Advanced

Examples

​Overview

​LogProcessor Trait

​SimpleLogProcessor

​Characteristics

​Usage

​When to Use

​Runtime Compatibility

​BatchLogProcessor

​Characteristics

​Basic Usage

​Configuration

​Configuration Parameters

​Export Triggers

​Example: Production Configuration

​Runtime Compatibility

​Force Flush

​Shutdown

​Dropped Logs

​Custom Log Processors

​event_enabled Optimization

​Processor Comparison

​Best Practices

​Troubleshooting

​Logs not exported

​High latency

​Memory usage too high

​Logs dropped

​See Also

log Appender

tracing Appender

Bridge API

Overview

Build docs developers (and LLMs) love

Overview

LogProcessor Trait

SimpleLogProcessor

Characteristics

Usage

When to Use

Runtime Compatibility

BatchLogProcessor

Characteristics

Basic Usage

Configuration

Configuration Parameters

Export Triggers

Example: Production Configuration

Runtime Compatibility

Force Flush

Shutdown

Dropped Logs

Custom Log Processors

event_enabled Optimization

Processor Comparison

Best Practices

Troubleshooting

Logs not exported

High latency

Memory usage too high

Logs dropped

See Also