Logging - CockroachDB

Overview

CockroachDB provides comprehensive logging capabilities for debugging, auditing, and monitoring. Logs can be written to files, stderr, or external systems, with fine-grained control over verbosity, channels, and formats.

Log Configuration

Basic Log Configuration

Configure logging using the --log flag:

cockroach start \
  --log='file-defaults: {dir: /var/log/cockroach}'

Log Configuration Format

The --log flag accepts YAML configuration:

--log='file-defaults: {dir: /var/log/cockroach, filter: INFO}'

Log Channels

CockroachDB organizes logs into channels based on their purpose:

Log Channels

Channel	Purpose	Examples
`DEV`	Development events	Debug information, internal state
`OPS`	Operational events	Node start/stop, configuration changes
`HEALTH`	Health checks	Liveness checks, node availability
`STORAGE`	Storage layer	Compaction, disk I/O, store operations
`SESSIONS`	SQL sessions	Connection events, authentication
`SQL_SCHEMA`	Schema changes	DDL operations, schema migrations
`SQL_EXEC`	SQL execution	Query execution, statement logs
`SQL_PERF`	SQL performance	Slow queries, execution statistics
`SQL_INTERNAL_PERF`	Internal SQL	Background SQL operations
`PRIVILEGES`	Security	Permission checks, role changes
`SENSITIVE_ACCESS`	Data access	Audit trail for sensitive data

Configuring Channels

Route specific channels to different outputs:

cockroach start --log='
  sinks:
    stderr:
      channels: [OPS, HEALTH]
      filter: INFO
    file-groups:
      security:
        channels: [PRIVILEGES, SENSITIVE_ACCESS]
        dir: /var/log/cockroach/security
        filter: INFO
      sql:
        channels: [SQL_EXEC, SQL_PERF]
        dir: /var/log/cockroach/sql
        filter: WARNING
'

Log Levels

CockroachDB supports standard log levels (verbosity):

Level	Severity	When to Use
`NONE`	Silent	Disable logging
`FATAL`	Critical errors	Process-terminating errors
`ERROR`	Errors	Errors that don’t terminate process
`WARNING`	Warnings	Potential issues
`INFO`	Information	Standard operational messages
`DEBUG`	Debug	Detailed debugging information

Setting Log Level

--log='file-defaults: {filter: WARNING}'

Log Destinations

File Logging

Default file logging configuration:

cockroach start --log='file-defaults: {dir: /var/log/cockroach}'

Log files are created in the specified directory:

/var/log/cockroach/
  cockroach.log         # Main log file
  cockroach-sql.log     # SQL execution logs
  cockroach-security.log # Security audit logs
  cockroach-health.log  # Health check logs

Log files automatically rotate when they reach the configured size limit (default 10MB).

Stderr Logging

Output logs to standard error:

cockroach start --log='sinks: {stderr: {channels: all}}'

Useful for:

Container deployments (Docker, Kubernetes)
Development environments
Systemd integration with journal

Syslog Integration

Send logs to syslog:

cockroach start --log='
  sinks:
    fluent-servers:
      syslog:
        channels: all
        address: localhost:514
        network: udp
'

Log File Management

Log Rotation

Configure automatic log rotation:

cockroach start --log='
  file-defaults:
    dir: /var/log/cockroach
    max-file-size: 50MiB
    max-group-size: 1GiB
'

Log rotation settings:

max-file-size: Maximum size of a single log file (default 10MB)
max-group-size: Maximum combined size for a log group (default 100MB)
Old files are renamed with timestamps
Oldest files deleted when group size exceeded

Log File Format

Log entries use a structured format:

I220303 15:42:17.123456 1234 5@util/log/file.go:456  [n1,s1] sample log message

Format breakdown:

I: Log level (I=INFO, W=WARNING, E=ERROR, F=FATAL)
220303: Date (YYMMDD)
15:42:17.123456: Time with microseconds
1234: Goroutine ID
5@util/log/file.go:456: Line number and source location
[n1,s1]: Node and store tags
sample log message: Log message

Viewing Logs

tail -f /var/log/cockroach/cockroach.log

Slow Query Logging

Enable logging for slow queries:

-- Set slow query threshold
SET CLUSTER SETTING sql.log.slow_query.latency_threshold = '100ms';

-- Enable slow query logging  
SET CLUSTER SETTING sql.log.slow_query.experimental_full_table_scans.enabled = true;

Slow queries are logged to the SQL_PERF channel:

cockroach start --log='
  sinks:
    file-groups:
      slow-queries:
        channels: SQL_PERF
        dir: /var/log/cockroach/slow
        filter: INFO
'

Slow Query Log Format

Slow query logs include:

Query text
Execution time
Number of rows processed
Resources used (CPU, memory, network)
Execution plan

Audit Logging

Enable audit logging for security compliance:

-- Enable audit logging for a table
ALTER TABLE sensitive_data SET (experimental_audit = 'READ_WRITE');

-- View audit events
SELECT * FROM system.eventlog 
WHERE "eventType" = 'sensitive_table_access'
ORDER BY timestamp DESC;

Configure audit log destination:

cockroach start --log='
  sinks:
    file-groups:
      audit:
        channels: SENSITIVE_ACCESS
        dir: /var/log/cockroach/audit
        filter: INFO
'

Audit logs can generate significant volume. Monitor disk space and configure appropriate retention policies.

Structured Logging

JSON Format

Output logs in JSON format for log aggregation systems:

cockroach start --log='
  file-defaults:
    format: json
    dir: /var/log/cockroach
'

JSON log entry example:

{
  "timestamp": "2022-03-03T15:42:17.123456Z",
  "severity": "INFO",
  "channel": "OPS",
  "goroutine": 1234,
  "file": "util/log/file.go",
  "line": 456,
  "tags": {"n": 1, "s": 1},
  "message": "sample log message"
}

Fluent Bit / Fluentd Integration

Configure CockroachDB to send logs to Fluent Bit:

cockroach start --log='
  sinks:
    fluent-servers:
      fluentbit:
        channels: all
        address: localhost:24224
        format: json-fluent-compact
'

Deprecated Log Flags

Older log flags are deprecated in favor of --log:

Deprecated Flags and Equivalents

Deprecated Flag	Modern Equivalent
`--log-dir`	`--log='file-defaults: {dir: /path}'`
`--log-file-verbosity`	`--log='file-defaults: {filter: LEVEL}'`
`--log-file-max-size`	`--log='file-defaults: {max-file-size: SIZE}'`
`--logtostderr`	`--log='sinks: {stderr: {channels: all}}'`

Checking Log Configuration

Verify your log configuration:

cockroach debug check-log-config --log='<your-config>'

This validates the log configuration without starting a node.

Common Log Configurations

Production Configuration

cockroach start --log='
  file-defaults:
    dir: /var/log/cockroach
    filter: INFO
    max-file-size: 50MiB
    max-group-size: 1GiB
  sinks:
    file-groups:
      security:
        channels: [PRIVILEGES, SENSITIVE_ACCESS]
        dir: /var/log/cockroach/security
      sql-slow:
        channels: SQL_PERF
        dir: /var/log/cockroach/slow
    stderr:
      channels: [OPS, HEALTH]
      filter: WARNING
'

Development Configuration

cockroach start --log='
  sinks:
    stderr:
      channels: all
      filter: DEBUG
'

Container Configuration

cockroach start --log='
  sinks:
    stderr:
      channels: all
      filter: INFO
      format: json
'

Log Analysis

Common Log Patterns

grep "^E" /var/log/cockroach/cockroach.log

Log Aggregation

Integrate with log aggregation systems:

ELK Stack: Use Filebeat to ship logs to Elasticsearch
Splunk: Configure Splunk forwarder to collect log files
CloudWatch Logs: Use CloudWatch agent for AWS deployments
Datadog: Send logs via Datadog agent

Best Practices

Logging Recommendations

Use appropriate log levels: INFO for production, DEBUG for troubleshooting
Separate sensitive logs: Isolate audit and security logs
Monitor log volume: High verbosity can impact performance
Rotate logs regularly: Prevent disk space exhaustion
Centralize logs: Use log aggregation for multi-node clusters
Retain audit logs: Comply with regulatory requirements
Test log configuration: Use debug check-log-config before deployment
Document log locations: Ensure team knows where to find logs

Troubleshooting

Logs Not Appearing

Verify log directory exists and is writable
Check --log configuration syntax
Review stderr for configuration errors
Ensure sufficient disk space

High Log Volume

Reduce log level (INFO → WARNING)
Disable verbose channels
Adjust max-file-size and retention
Filter out noisy log messages

Missing Log Entries

Check log level filters
Verify channel configuration
Review log rotation settings
Check for disk space issues

Getting Started

Architecture

SQL Reference

Administration

Operations

Performance

​Overview

​Log Configuration

​Basic Log Configuration

​Log Configuration Format

​Log Channels

​Configuring Channels

​Log Levels

​Setting Log Level

​Log Destinations

​File Logging

​Stderr Logging

​Syslog Integration

​Log File Management

​Log Rotation

​Log File Format

​Viewing Logs

​Slow Query Logging

​Slow Query Log Format

​Audit Logging

​Structured Logging

​JSON Format

​Fluent Bit / Fluentd Integration

​Deprecated Log Flags

​Checking Log Configuration

​Common Log Configurations

​Production Configuration

​Development Configuration

​Container Configuration

​Log Analysis

​Common Log Patterns

​Log Aggregation

​Best Practices

​Troubleshooting

​Logs Not Appearing

​High Log Volume

​Missing Log Entries

​See Also

Build docs developers (and LLMs) love

Overview

Log Configuration

Basic Log Configuration

Log Configuration Format

Log Channels

Configuring Channels

Log Levels

Setting Log Level

Log Destinations

File Logging

Stderr Logging

Syslog Integration

Log File Management

Log Rotation

Log File Format

Viewing Logs

Slow Query Logging

Slow Query Log Format

Audit Logging

Structured Logging

JSON Format

Fluent Bit / Fluentd Integration

Deprecated Log Flags

Checking Log Configuration

Common Log Configurations

Production Configuration

Development Configuration

Container Configuration

Log Analysis

Common Log Patterns

Log Aggregation

Best Practices

Troubleshooting

Logs Not Appearing

High Log Volume

Missing Log Entries

See Also