Skip to main content

​
Overview

Snuba uses a migration system to manage ClickHouse database schema changes. The migration commands provide tools for listing, running, and reverting migrations across different storage groups. Migrations are organized into migration groups that correspond to different parts of the system (e.g., events, transactions, system). Each migration group can have its own readiness state that determines when it should be applied.

​
Main Command

snuba migrations [SUBCOMMAND] [OPTIONS]
The migrations command provides several subcommands for different operations.

​
Subcommands

​
list

Lists all migrations and their current status across all migration groups. 
snuba migrations list
This command displays:
  • All migration groups with their readiness states
  • Migration IDs within each group
  • Status indicators:
    • [X] - Completed
    • [ ] - Not started
    • [-] - In progress
  • Blocking status for incomplete migrations
  • Warnings for migrations that no longer exist in code

​
Example Output

events (readiness_state: complete)
[X]  0001_events_initial
[X]  0002_events_onpremise_compatibility
[X]  0003_errors
[ ]  0004_add_new_column (blocking)

transactions (readiness_state: limited)
[X]  0001_transactions
[X]  0002_transactions_onpremise_compatibility

system (readiness_state: complete)
[X]  0001_migrations

​
migrate

Runs pending migrations for a specific group or all groups. 
snuba migrations migrate [OPTIONS] [THROUGH]
​
THROUGH
string
default:"all"
Run migrations up to and including this migration ID. Use "all" to run all pending migrations.
​
--group
string
Migration group to run migrations for. If not specified, runs migrations for all groups.Common groups:
  • system - Core system tables
  • events - Error events storage
  • transactions - Transaction events storage
  • outcomes - Outcomes/billing data
  • sessions - Session data
  • metrics - Metrics storage
​
--readiness-state
choice
Only run migrations for groups with specific readiness states. Can be specified multiple times.Available states:
  • complete - Fully ready for production
  • limited - Limited availability
  • experimental - Experimental features
  • partial - Partially implemented
​
--force
flag
Force running of blocking migrations. Required for migrations marked as blocking.
​
--fake
flag
Mark migrations as completed without actually running them. Useful for fixing migration state.
​
--check-dangerous
flag
Check for dangerous operations (like dropping tables or columns) and require confirmation.
​
--log-level
choice
Logging level (critical, error, warning, info, debug, notset).

​
Examples

# Run all pending migrations across all groups
snuba migrations migrate --force
Blocking migrations require the --force flag as they may contain dangerous operations or require downtime. Always review blocking migrations before applying.

​
run

Runs a single specific migration. 
snuba migrations run [OPTIONS]
​
--group
string
required
Migration group containing the migration.
​
--migration-id
string
required
ID of the migration to run.
​
--force
flag
Required to run blocking migrations.
​
--fake
flag
Mark migration as completed without running. Requires confirmation unless --yes is also passed.
​
--dry-run
flag
Show what SQL would be executed without actually running it.
​
--yes
flag
Skip confirmation prompts (use with caution).
​
--check-dangerous
flag
Check for dangerous operations and require confirmation.
​
--log-level
choice
Logging level to use.

​
Examples

# Run a specific migration
snuba migrations run \
  --group events \
  --migration-id 0004_add_new_column \
  --force
Migrations that are already in progress or completed will not be run again. Use --fake to fix migration state if needed.

​
revert

Reverses migrations for a group, running them backwards. 
snuba migrations revert [OPTIONS] [THROUGH]
​
THROUGH
string
default:"all"
Revert migrations back to and including this migration ID. Use "all" to revert all migrations.
​
--group
string
Migration group to revert migrations for. If not specified, reverts migrations for all groups.
​
--readiness-state
choice
Only revert migrations for groups with specific readiness states.
​
--force
flag
required
Required to perform revert operations.
​
--fake
flag
Mark migrations as reverted without actually running the reverse operations.
​
--include-system
flag
Also revert SYSTEM migrations. By default, system migrations are NOT reversed.
​
--log-level
choice
Logging level to use.

​
Examples

# Revert all migrations in events group
snuba migrations revert --group events --force
Reverting migrations can result in data loss. Always backup data before reverting migrations. The --force flag is required to prevent accidental reverts.

​
reverse

Reverses a single specific migration. 
snuba migrations reverse [OPTIONS]
​
--group
string
required
Migration group containing the migration.
​
--migration-id
string
required
ID of the migration to reverse.
​
--force
flag
required
Required to reverse an already completed migration.
​
--fake
flag
Mark migration as reversed without running. Requires confirmation unless --yes is passed.
​
--dry-run
flag
Show what SQL would be executed without running it.
​
--yes
flag
Skip confirmation prompts.
​
--log-level
choice
Logging level to use.

​
Examples

# Reverse a specific migration
snuba migrations reverse \
  --group events \
  --migration-id 0004_add_new_column \
  --force

​
reverse-in-progress

Reverses any migrations currently in an “in progress” state. 
snuba migrations reverse-in-progress [OPTIONS]
​
--group
string
Only reverse in-progress migrations for this group. If not specified, checks all groups.
​
--fake
flag
Mark migrations as reversed without running. Requires confirmation unless --yes is passed.
​
--dry-run
flag
Show which migrations would be reversed without executing.
​
--yes
flag
Skip confirmation prompts.
​
--log-level
choice
Logging level to use.

​
Examples

# Reverse all in-progress migrations
snuba migrations reverse-in-progress
This command is useful for recovering from failed migrations that left the system in an in-progress state.

​
generate

Generates a new migration file based on schema changes in a storage definition. 
snuba migrations generate STORAGE_PATH [OPTIONS]
​
STORAGE_PATH
string
required
Path to the modified storage YAML definition file.Must match pattern: snuba/datasets/configuration/.*/storages/.*.(yml|yaml)
​
--name
string
Optional name for the generated migration. If not provided, a default name is generated.

​
How It Works

  1. Modify Storage YAML - Edit the storage definition file to add/modify schema
  2. Leave Uncommitted - The changes must be uncommitted in your working directory
  3. Run Generate - Command analyzes the diff between HEAD and working directory
  4. Migration Created - A Python migration file is generated in the appropriate directory
Currently only column addition is supported for automatic generation. Other schema changes require manual migration creation.

​
Examples

# Generate migration for modified storage
snuba migrations generate \
  snuba/datasets/configuration/events/storages/errors.yaml
The migration is generated based on uncommitted changes. Ensure your storage modifications are correct before generating the migration.

​
Migration Groups and Readiness States

​
Common Migration Groups

  • system - Core system tables for storing migration status
  • events - Error and exception events
  • transactions - Transaction/span events
  • outcomes - Outcome and billing data
  • sessions - Session aggregates
  • metrics - Metrics data (various types)
  • querylog - Query logs and performance data
  • profiles - Profiling data
  • replays - Session replay data

​
Readiness States

Fully implemented and production-ready. Safe to run in all environments.
Limited availability or partial rollout. May be enabled only for certain tenants.
Experimental features under active development. Use with caution.
Partially implemented functionality. May be missing features or optimizations.

​
Migration Status

Migrations can be in one of three states:
  1. NOT_STARTED - Migration has not been applied
  2. IN_PROGRESS - Migration is currently running (or was interrupted)
  3. COMPLETED - Migration was successfully applied

​
Best Practices

​
Before Running Migrations

1

Review Migration Code

Always review the migration code before running, especially for blocking migrations.
2

Use Dry Run

Use --dry-run to preview SQL that will be executed.
snuba migrations run --group events --migration-id 0005 --dry-run
3

Check for Blocking Operations

Use --check-dangerous to identify potentially dangerous operations.
snuba migrations run --group events --migration-id 0005 --check-dangerous
4

Backup Data

Ensure you have backups before running destructive migrations.
5

Test in Staging

Always test migrations in a staging environment first.

​
Running Migrations in Production

# 1. List current status
snuba migrations list

# 2. Review specific migration
snuba migrations run \
  --group events \
  --migration-id 0005_new_column \
  --dry-run

# 3. Check for dangerous operations
snuba migrations run \
  --group events \
  --migration-id 0005_new_column \
  --check-dangerous

# 4. Run with verbose logging
snuba migrations run \
  --group events \
  --migration-id 0005_new_column \
  --force \
  --log-level debug

​
Handling Failed Migrations

If a migration is interrupted:
# Option 1: Reverse the in-progress migration
snuba migrations reverse-in-progress --group events

# Option 2: Fake reverse if safe
snuba migrations reverse-in-progress --group events --fake --yes

# Then retry the migration
snuba migrations run --group events --migration-id 0005 --force
If you manually applied schema changes:
# Mark migration as complete without running
snuba migrations run \
  --group events \
  --migration-id 0005 \
  --fake
To rollback a completed migration:
# Reverse the specific migration
snuba migrations reverse \
  --group events \
  --migration-id 0005 \
  --force

​
Connection Requirements

All migration commands (except generate with --dry-run) check ClickHouse connections before executing. Ensure ClickHouse clusters are accessible before running migrations.
The system checks connections to:
  • All clusters (for migrate and revert without specific group)
  • Specific clusters for the migration group (when --group is specified)
  • Clusters matching readiness states (when --readiness-state is used)

​
Monitoring Migrations

​
Check Migration Status

# List all migrations and their status
snuba migrations list

# Check specific group
snuba migrations list | grep "events"

# Check for in-progress migrations
snuba migrations list | grep "IN PROGRESS"

​
Verbose Logging

Use debug logging to monitor migration execution: 
snuba migrations migrate \
  --group events \
  --log-level debug \
  --force

​
Common Workflows

​
Initial Setup

# Run all system migrations first
snuba migrations migrate --group system --force

# Then run all other migrations
snuba migrations migrate --force

​
Deploying New Features

# Run migrations for specific readiness level
snuba migrations migrate \
  --readiness-state limited \
  --force

​
Developing New Migrations

# 1. Modify storage YAML
vim snuba/datasets/configuration/events/storages/errors.yaml

# 2. Generate migration
snuba migrations generate \
  snuba/datasets/configuration/events/storages/errors.yaml \
  --name add_new_column

# 3. Test migration locally
snuba migrations run \
  --group events \
  --migration-id 0XXX_add_new_column \
  --dry-run

# 4. Apply migration
snuba migrations run \
  --group events \
  --migration-id 0XXX_add_new_column \
  --force

​
See Also

  • MIGRATIONS.md in the repository root for detailed migration development guide
  • Migration Runner source code: snuba/migrations/runner.py
  • Migration Groups definition: snuba/migrations/groups.py

Build docs developers (and LLMs) love

Get started for freeTalk to us