cockroach debug - CockroachDB

The cockroach debug command provides advanced tools for debugging and troubleshooting CockroachDB clusters.

Debug commands are advanced tools intended for troubleshooting by experienced operators. Many commands operate directly on storage files and should be used with caution.

Synopsis

cockroach debug [command] [flags]

Description

Debug commands help diagnose issues with CockroachDB clusters, inspect internal state, and extract data from failed nodes. These commands are particularly useful when:

A node has failed and cannot restart
Investigating data corruption or consistency issues
Troubleshooting performance problems
Extracting data from offline stores
Analyzing cluster internals

Key Subcommands

debug zip

Collect diagnostic information from all nodes in a cluster.

cockroach debug zip <output-file> [flags]

Creates a ZIP archive containing:

Cluster status and configuration
Node logs and stack traces
Range distribution and replica status
Schema and table statistics
Recent query activity
System metrics and events

debug zip is the primary tool for gathering information when reporting bugs or troubleshooting issues.

Flags

--include-range-info

boolean

default:"false"

Include detailed range replica information (can be large).

--exclude-nodes

string

Comma-separated list of node IDs to exclude from collection.

--timeout

duration

default:"2m"

Maximum time to wait for data collection per node.

Examples

cockroach debug zip debug.zip --host=localhost:26257 --insecure

debug keys

Dump all keys in a store’s storage engine.

cockroach debug keys <store-directory> [flags]

This command reads from storage files directly. The node must be stopped before running this command.

Flags

--values

boolean

default:"false"

Print key values in addition to keys.

--sizes

boolean

default:"false"

Print key and value sizes.

--decode-as-table-desc

string

Base64-encoded table descriptor for decoding row data.

Examples

# List all keys
cockroach debug keys /path/to/cockroach-data

# List keys with values
cockroach debug keys /path/to/cockroach-data --values

# Show key sizes
cockroach debug keys /path/to/cockroach-data --sizes

debug range-data

Dump all data for a specific range.

cockroach debug range-data <store-directory> <range-id> [flags]

Use debug range-descriptors first to find range IDs.

Flags

--replicated

boolean

default:"false"

Show only replicated data (excludes local Raft state).

Examples

# Show all data for range 42
cockroach debug range-data /path/to/cockroach-data 42

# Show only replicated data
cockroach debug range-data /path/to/cockroach-data 42 --replicated

debug range-descriptors

Print all range descriptors in a store.

cockroach debug range-descriptors <store-directory>

Shows range boundaries, replica locations, and range metadata. Example:

cockroach debug range-descriptors /path/to/cockroach-data

debug raft-log

Print the Raft log for a specific range.

cockroach debug raft-log <store-directory> <range-id>

Displays all Raft log entries for investigating consensus issues. Example:

cockroach debug raft-log /path/to/cockroach-data 42

debug estimate-gc

Estimate garbage collection impact without performing it.

cockroach debug estimate-gc <store-directory> [range-id] [ttl-seconds] [intent-age]

Helps predict GC impact before running actual garbage collection.

Arguments

store-directory - Path to node’s data directory (required)
range-id - Specific range ID to analyze (optional, defaults to all)
ttl-seconds - GC TTL in seconds (optional, defaults to 86400 = 24 hours)
intent-age - Intent age threshold as duration (optional, defaults to 2h)

Examples

cockroach debug estimate-gc /path/to/cockroach-data

debug compact

Manually compact storage engine SSTables.

cockroach debug compact <store-directory> [flags]

Stop the node before running compaction. This operation can take significant time for large stores.

Flags

--max-concurrency

int

default:"GOMAXPROCS"

Maximum concurrent compaction operations.

Example

cockroach debug compact /path/to/cockroach-data --max-concurrency=4

debug check-store

Run consistency checks on store data files.

cockroach debug check-store <store-directory> [flags]

Verifies:

SSTable integrity
Manifest consistency
Key ordering
File checksums

Example:

cockroach debug check-store /path/to/cockroach-data

debug ballast

Create a ballast file to reserve disk space.

cockroach debug ballast <file-path> [flags]

Ballast files can be deleted in emergencies to free disk space, allowing the node to continue operating.

Flags

--size

string

Size of ballast file. Accepts bytes, size suffixes (e.g., 1GB), or percentage of disk.

Examples

cockroach debug ballast /path/to/ballast.txt --size=10GB

debug decode-key

Decode hex or base64-encoded keys.

cockroach debug decode-key <encoded-key> [flags]

Flags

--encoding

string

default:"hex"

Key encoding format: hex or base64.

--user-key

boolean

default:"false"

Decode as user key (without MVCC timestamp).

Examples

cockroach debug decode-key BB89F902ADB43000151C2D1ED07DE6C009

debug decode-value

Decode hex-encoded key-value pairs.

cockroach debug decode-value <hex-key> <hex-value>

Displays the decoded and pretty-printed value. Example:

cockroach debug decode-value BB89F902ADB4 0A0673797374656D

debug merge-logs

Merge log files from multiple nodes into a single stream.

cockroach debug merge-logs <log-file-pattern>... [flags]

Extremely useful for analyzing distributed behavior across nodes.

Flags

--from

timestamp

Filter messages before this time.

--to

timestamp

Filter messages after this time.

--filter

regex

Regular expression to filter log messages.

--program-filter

regex

Filter by program name (cockroach, cockroach-sql, etc.).

Examples

cockroach debug merge-logs '/var/log/cockroach/*.log'

debug gossip-values

Dump gossip network values from a running node.

cockroach debug gossip-values [flags]

Shows cluster-wide gossip information including:

Node descriptors
Store descriptors
Liveness records
Cluster ID

Example:

cockroach debug gossip-values --host=localhost:26257 --insecure

debug list-files

List files managed by the storage engine.

cockroach debug list-files <store-directory> [flags]

Shows SSTables, WAL files, manifest, and metadata files. Example:

cockroach debug list-files /path/to/cockroach-data

Pebble Tool Integration

CockroachDB includes the Pebble storage engine debugging tool:

cockroach debug pebble [command]

Supports all Pebble tool commands for low-level storage inspection:

db - Database introspection
manifest - Manifest file inspection
sstable - SSTable analysis
wal - Write-ahead log inspection

Example:

# Inspect manifest
cockroach debug pebble manifest dump /path/to/MANIFEST-000001

# Analyze SSTable
cockroach debug pebble sstable scan /path/to/000123.sst

Connection Flags

Commands that connect to running clusters support:

--host

string

default:"localhost:26257"

Node to connect to.

--certs-dir

string

Certificate directory for secure connections.

--insecure

boolean

default:"false"

Connect without encryption.

Common Use Cases

Investigating Failed Node

# 1. Check store integrity
cockroach debug check-store /mnt/cockroach-data

# 2. List range descriptors
cockroach debug range-descriptors /mnt/cockroach-data

# 3. Examine specific range
cockroach debug range-data /mnt/cockroach-data 42

Troubleshooting Performance

# 1. Collect cluster diagnostics
cockroach debug zip debug.zip --include-range-info --insecure

# 2. Analyze GC impact
cockroach debug estimate-gc /mnt/cockroach-data

# 3. Check gossip state
cockroach debug gossip-values --insecure

Analyzing Distributed Issues

# Merge logs from all nodes
cockroach debug merge-logs '/var/log/node*/cockroach.log' \
  --filter='range.*42' \
  --from='2024-03-01 10:00:00'

Best Practices

Important Safety Guidelines:

Always stop the node before running commands that access storage files directly
Make backups before running any commands that modify storage
Test on non-production clusters first when possible
Consult documentation or support before using unfamiliar debug commands
Never modify raw storage files manually

When to Use Debug Commands:

Node won’t start and logs don’t provide sufficient information
Investigating suspected data corruption
Analyzing performance issues with storage layer
Extracting data from failed clusters
Working with Cockroach Labs support on complex issues

Troubleshooting

Cannot Open Store

Ensure node is stopped
Verify you have read permissions on data directory
Check that encryption keys are available (if using encryption)
Ensure sufficient disk space

Encryption Errors

If using encrypted stores, ensure encryption keys are properly configured:

export COCKROACH_ENCRYPTION_KEY="path/to/key"
cockroach debug keys /path/to/encrypted-store

Contributing

CLI Reference

​Synopsis

​Description

​Key Subcommands

​debug zip

​Flags

​Examples

​debug keys

​Flags

​Examples

​debug range-data

​Flags

​Examples

​debug range-descriptors

​debug raft-log

​debug estimate-gc

​Arguments

​Examples

​debug compact

​Flags

​Example

​debug check-store

​debug ballast

​Flags

​Examples

​debug decode-key

​Flags

​Examples

​debug decode-value

​debug merge-logs

​Flags

​Examples

​debug gossip-values

​debug list-files

​Pebble Tool Integration

​Connection Flags

​Common Use Cases

​Investigating Failed Node

​Troubleshooting Performance

​Analyzing Distributed Issues

​Best Practices

​Troubleshooting

​Cannot Open Store

​Encryption Errors

​See Also

Build docs developers (and LLMs) love

Synopsis

Description

Key Subcommands

debug zip

Flags

Examples

debug keys

Flags

Examples

debug range-data

Flags

Examples

debug range-descriptors

debug raft-log

debug estimate-gc

Arguments

Examples

debug compact

Flags

Example

debug check-store

debug ballast

Flags

Examples

debug decode-key

Flags

Examples

debug decode-value

debug merge-logs

Flags

Examples

debug gossip-values

debug list-files

Pebble Tool Integration

Connection Flags

Common Use Cases

Investigating Failed Node

Troubleshooting Performance

Analyzing Distributed Issues

Best Practices

Troubleshooting

Cannot Open Store

Encryption Errors

See Also