Creating Migrations

Overview

This guide covers how to create migrations in Snuba, from auto-generation for simple changes to writing custom migrations for complex scenarios.

Auto-Generating Migrations

For common use cases like adding columns, Snuba can auto-generate migrations from storage configuration changes.

Locate the storage configuration

Find the relevant storage.yaml file for your table:

# Storage configurations are located in:
snuba/datasets/configuration/<dataset>/storages/<storage>.yaml

# Example:
snuba/datasets/configuration/group_attributes/storages/group_attributes.yaml

Modify the storage schema

Edit the schema.columns section to add your new column:

schema:
  columns:
    - name: project_id
      type: UInt(64)
    - name: group_id
      type: UInt(64)
    - name: new_column  # Add new column
      type: String
      args:
        nullable: true

Generate the migration

Run the migration generator:

snuba migrations generate path/to/storage.yaml

This creates a new migration file with the appropriate operations.

Review and commit

Review the generated migration file, test it, and commit:

# Test the migration
snuba migrations run --group <group> --migration-id <id> --dry-run

# Commit if everything looks good
git add snuba/snuba_migrations/<group>/<migration_id>.py
git commit -m "Add new_column to group_attributes"

Auto-generation currently supports adding columns to existing tables. For other operations, you’ll need to write a custom migration.

Writing Custom Migrations

Step 1: Determine the Migration Group

Choose the appropriate migration group based on which tables you’re modifying:

Group	Purpose
`system`	Core migration infrastructure
`events`	Error event tables
`transactions`	Transaction tables
`discover`	Discover dataset
`metrics`	Metrics and counters
`sessions`	Session data
`profiles`	Profiling data
`functions`	Function profiling
`spans`	Span data

Step 2: Create the Migration File

Migrations must follow a specific naming convention:

# Format: <4-digit-number>_<descriptive_name>.py
# Number must be sequential within the group

# Find the next migration number
ls snuba/snuba_migrations/<group>/ | grep -E '^[0-9]{4}' | tail -1

# Create new migration file
touch snuba/snuba_migrations/<group>/0042_add_user_agent_column.py

Step 3: Choose Migration Type

Select the appropriate base class for your migration:

ClickhouseNodeMigration (SQL Operations)

For DDL operations like creating tables, adding columns, or modifying indexes:

snuba/snuba_migrations/events/0042_add_user_agent_column.py

from typing import Sequence
from snuba.clickhouse.columns import Column, String
from snuba.clusters.storage_sets import StorageSetKey
from snuba.migrations import migration, operations
from snuba.migrations.columns import MigrationModifiers as Modifiers
from snuba.migrations.operations import OperationTarget

class Migration(migration.ClickhouseNodeMigration):
    blocking = False
    
    def forwards_ops(self) -> Sequence[operations.SqlOperation]:
        return [
            operations.AddColumn(
                storage_set=StorageSetKey.EVENTS,
                table_name="errors_local",
                column=Column(
                    "user_agent",
                    String(Modifiers(nullable=True))
                ),
                after="user_email",
                target=OperationTarget.LOCAL,
            ),
            operations.AddColumn(
                storage_set=StorageSetKey.EVENTS,
                table_name="errors_dist",
                column=Column(
                    "user_agent",
                    String(Modifiers(nullable=True))
                ),
                after="user_email",
                target=OperationTarget.DISTRIBUTED,
            ),
        ]
    
    def backwards_ops(self) -> Sequence[operations.SqlOperation]:
        return [
            operations.DropColumn(
                storage_set=StorageSetKey.EVENTS,
                table_name="errors_dist",
                column_name="user_agent",
                target=OperationTarget.DISTRIBUTED,
            ),
            operations.DropColumn(
                storage_set=StorageSetKey.EVENTS,
                table_name="errors_local",
                column_name="user_agent",
                target=OperationTarget.LOCAL,
            ),
        ]

CodeMigration (Python Functions)

For complex logic, data migrations, or operations requiring runtime decisions:

snuba/snuba_migrations/events/0043_migrate_user_data.py

import logging
from typing import Sequence
from snuba.clusters.cluster import ClickhouseClientSettings, get_cluster
from snuba.clusters.storage_sets import StorageSetKey
from snuba.migrations import migration, operations

def migrate_user_data(logger: logging.Logger) -> None:
    """Migrate user data from old format to new format."""
    cluster = get_cluster(StorageSetKey.EVENTS)
    connection = cluster.get_query_connection(
        ClickhouseClientSettings.MIGRATE
    )
    
    # Perform data migration
    logger.info("Starting user data migration")
    connection.execute(
        "ALTER TABLE errors_local UPDATE user_name = ... WHERE ..."
    )
    logger.info("User data migration complete")

class Migration(migration.CodeMigration):
    blocking = True  # Data migrations should be blocking
    
    def forwards_global(self) -> Sequence[operations.GenericOperation]:
        return [
            operations.RunPython(
                func=migrate_user_data,
                description="Migrate user data to new format",
            ),
        ]
    
    def backwards_global(self) -> Sequence[operations.GenericOperation]:
        # Usually cannot reverse data migrations
        return []

Available Operations

Table Operations

operations.CreateTable(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    columns=[
        Column("project_id", UInt(64)),
        Column("timestamp", DateTime()),
        Column("event_id", UUID()),
    ],
    engine=table_engines.MergeTree(
        storage_set=StorageSetKey.EVENTS,
        order_by="(project_id, timestamp)",
        partition_by="(toMonday(timestamp))",
        ttl="timestamp + toIntervalDay(90)",
    ),
    target=OperationTarget.LOCAL,
)

Column Operations

operations.AddColumn(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    column=Column(
        "new_field",
        String(Modifiers(nullable=True, low_cardinality=True))
    ),
    after="existing_column",  # Optional positioning
    target=OperationTarget.LOCAL,
)

Index Operations

operations.AddIndex(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    index_name="user_id_bloom",
    index_expression="user_id",
    index_type="bloom_filter(0.01)",
    granularity=4,
    target=OperationTarget.LOCAL,
)

TTL Operations

operations.ModifyTableTTL(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    reference_column="timestamp",
    ttl_days=90,
    materialize_ttl_on_modify=True,
    target=OperationTarget.LOCAL,
)

Settings Operations

operations.ModifyTableSettings(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    settings={
        "index_granularity": 8192,
        "min_bytes_for_wide_part": 0,
    },
    target=OperationTarget.LOCAL,
)

Custom SQL

RunSql

operations.RunSql(
    storage_set=StorageSetKey.EVENTS,
    statement="""
        ALTER TABLE events_local
        MODIFY SETTING max_concurrent_queries = 100
    """,
    target=OperationTarget.LOCAL,
)

Data Operations

InsertIntoSelect

operations.InsertIntoSelect(
    storage_set=StorageSetKey.EVENTS,
    dest_table_name="events_new_local",
    dest_columns=["project_id", "timestamp", "event_id"],
    src_table_name="events_old_local",
    src_columns=["project_id", "timestamp", "event_id"],
    where="timestamp >= '2024-01-01'",
    target=OperationTarget.LOCAL,
)

Operation Ordering

The order of operations matters, especially for distributed tables:

Adding Columns or Tables

def forwards_ops(self) -> Sequence[operations.SqlOperation]:
    return [
        # 1. Create/modify local tables FIRST
        operations.AddColumn(
            table_name="events_local",
            target=OperationTarget.LOCAL,
            # ...
        ),
        # 2. Then create/modify distributed tables
        operations.AddColumn(
            table_name="events_dist",
            target=OperationTarget.DISTRIBUTED,
            # ...
        ),
    ]

Add to local tables first, then distributed. This prevents the distributed table from referencing non-existent columns.

Dropping Columns or Tables

def backwards_ops(self) -> Sequence[operations.SqlOperation]:
    return [
        # 1. Drop from distributed tables FIRST
        operations.DropColumn(
            table_name="events_dist",
            target=OperationTarget.DISTRIBUTED,
            # ...
        ),
        # 2. Then drop from local tables
        operations.DropColumn(
            table_name="events_local",
            target=OperationTarget.LOCAL,
            # ...
        ),
    ]

Drop from distributed tables first, then local. Dropping local columns while the distributed table references them causes errors.

Column Types and Modifiers

Snuba provides type-safe column definitions:

from snuba.clickhouse.columns import (
    Column, String, UInt, UUID, DateTime, Array, Nested
)
from snuba.migrations.columns import MigrationModifiers as Modifiers

columns = [
    # Basic types
    Column("id", UInt(64)),
    Column("name", String()),
    Column("timestamp", DateTime()),
    Column("event_id", UUID()),
    
    # With modifiers
    Column("environment", String(Modifiers(
        nullable=True,
        low_cardinality=True,
    ))),
    
    # Arrays
    Column("tag_values", Array(String())),
    
    # Nested structures
    Column("tags", Nested([
        ("key", String()),
        ("value", String()),
    ])),
]

Table Engines

Snuba automatically selects the appropriate engine based on cluster configuration:

table_engines.MergeTree(
    storage_set=StorageSetKey.EVENTS,
    order_by="(project_id, timestamp)",
    primary_key="(project_id, timestamp)",  # Optional
    partition_by="(toMonday(timestamp))",
    sample_by="cityHash64(event_id)",  # Optional
    ttl="timestamp + toIntervalDay(90)",
    settings={
        "index_granularity": 8192,
    },
)

In single-node mode, MergeTree engines are used. In distributed mode, they automatically become ReplicatedMergeTree with appropriate ZooKeeper paths.

Testing Migrations

Dry Run

Preview SQL without executing:

snuba migrations run \
    --group events \
    --migration-id 0042_add_user_agent_column \
    --dry-run

Output:

Local op: ALTER TABLE errors_local ADD COLUMN IF NOT EXISTS user_agent String AFTER user_email;
Dist op: ALTER TABLE errors_dist ADD COLUMN IF NOT EXISTS user_agent String AFTER user_email;

Run Migration

Execute the migration:

snuba migrations run \
    --group events \
    --migration-id 0042_add_user_agent_column \
    --force

Verify Results

Check migration status:

snuba migrations list --group events

Verify schema changes:

echo "DESCRIBE TABLE errors_local" | clickhouse-client

Best Practices

Test locally first

Always test migrations in development before production:

# Test with dry-run
snuba migrations run --group <group> --migration-id <id> --dry-run

# Execute in local environment
snuba migrations run --group <group> --migration-id <id> --force

Write reversible migrations

Always provide backward operations that restore the original state:

def backwards_ops(self) -> Sequence[operations.SqlOperation]:
    # Don't just return empty list
    return [
        operations.DropColumn(
            table_name="events_local",
            column_name="user_agent",
            target=OperationTarget.LOCAL,
        ),
    ]

Set blocking flag appropriately

Mark migrations as blocking if they:

Migrate large amounts of data
Require significant processing time
Need consumers to be stopped

class Migration(migration.ClickhouseNodeMigration):
    blocking = True  # Requires --force

Update storage schemas

After adding a migration, update the corresponding storage schema file to match. Tests will fail if schemas are inconsistent.

Consider operation order

Use the validator to catch ordering issues:

# tests/migrations/test_validator.py
from snuba.migrations.validator import validate_migration_order

validate_migration_order(migration)

Common Pitfalls

Don’t forget operation targets

# BAD - target defaults to UNSET
operations.AddColumn(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    column=Column("field", String()),
)

# GOOD - explicitly set target
operations.AddColumn(
    storage_set=StorageSetKey.EVENTS,
    table_name="events_local",
    column=Column("field", String()),
    target=OperationTarget.LOCAL,  # Explicit target
)

Wrong operation order

# BAD - distributed before local
def forwards_ops(self):
    return [
        operations.AddColumn(target=OperationTarget.DISTRIBUTED, ...),
        operations.AddColumn(target=OperationTarget.LOCAL, ...),
    ]

# GOOD - local before distributed
def forwards_ops(self):
    return [
        operations.AddColumn(target=OperationTarget.LOCAL, ...),
        operations.AddColumn(target=OperationTarget.DISTRIBUTED, ...),
    ]

Dropping primary key columnsYou cannot drop columns that are part of the primary key or sorting key:

# This will fail if project_id is in ORDER BY
operations.DropColumn(
    table_name="events_local",
    column_name="project_id",
    target=OperationTarget.LOCAL,
)

Get Started

Architecture

Query Languages

Datasets

Configuration

Migrations

Operations

Creating Migrations

Overview

Auto-Generating Migrations

Writing Custom Migrations

Step 1: Determine the Migration Group

Step 2: Create the Migration File

Step 3: Choose Migration Type

ClickhouseNodeMigration (SQL Operations)

CodeMigration (Python Functions)

Available Operations

Table Operations

Column Operations

Index Operations

TTL Operations

Settings Operations

Custom SQL

Data Operations

Operation Ordering

Adding Columns or Tables

Dropping Columns or Tables

Column Types and Modifiers

Table Engines

Testing Migrations

Dry Run

Run Migration

Verify Results

Best Practices

Common Pitfalls

Next Steps

Migration Modes

Distributed Strategies

Build docs developers (and LLMs) love

Get Started

Architecture

Query Languages

Datasets

Configuration

Migrations

Operations

​Overview

​Auto-Generating Migrations

​Writing Custom Migrations

​Step 1: Determine the Migration Group

​Step 2: Create the Migration File

​Step 3: Choose Migration Type

​ClickhouseNodeMigration (SQL Operations)

​CodeMigration (Python Functions)

​Available Operations

​Table Operations

​Column Operations

​Index Operations

​TTL Operations

​Settings Operations

​Custom SQL

​Data Operations

​Operation Ordering

​Adding Columns or Tables

​Dropping Columns or Tables

​Column Types and Modifiers

​Table Engines

​Testing Migrations

​Dry Run

​Run Migration

​Verify Results

​Best Practices

​Common Pitfalls

​Next Steps

Migration Modes

Distributed Strategies

Build docs developers (and LLMs) love

Overview

Auto-Generating Migrations

Writing Custom Migrations

Step 1: Determine the Migration Group

Step 2: Create the Migration File

Step 3: Choose Migration Type

ClickhouseNodeMigration (SQL Operations)

CodeMigration (Python Functions)

Available Operations

Table Operations

Column Operations

Index Operations

TTL Operations

Settings Operations

Custom SQL

Data Operations

Operation Ordering

Adding Columns or Tables

Dropping Columns or Tables

Column Types and Modifiers

Table Engines

Testing Migrations

Dry Run

Run Migration

Verify Results

Best Practices

Common Pitfalls

Next Steps