Installation

Overview

Snuba can be installed and run in several ways depending on your use case. This guide covers:

Devservices - Recommended for local development (easiest)
Docker Compose - Containerized deployment
Local Development - Source installation with native dependencies
Production Deployment - Production-ready configuration

Prerequisites

Before installing Snuba, ensure you have the following dependencies:

All Methods
Docker
Local Development

Required Services:

ClickHouse 25.3+
Apache Kafka (with Zookeeper)
Redis 4.5+

System Requirements:

8GB+ RAM (16GB recommended)
4+ CPU cores
20GB+ disk space

Method 1: Devservices (Recommended)

Devservices is the easiest way to get started with Snuba development. It automatically manages all service dependencies.

Install Devservices

Install the devservices package:

pip install devservices>=1.2.1

Devservices requires Docker to be installed and running.

Clone Snuba Repository

Clone the Snuba repository and navigate to it:

git clone https://github.com/getsentry/snuba.git
cd snuba

Install Snuba Dependencies

Install Snuba Python dependencies using uv or pip:

# Install uv if not already installed
pip install uv

# Install dependencies
uv sync

Build Rust Components (Optional)

If you need the Rust consumer functionality:

# Build rust_snuba module
make install-rs-dev

This requires:

Rust toolchain (install from rustup.rs)
maturin for building Python extensions

Skip this step if you only need Python components. The Rust consumer is optional but recommended for better performance.

Start Services with Devservices

Start required services (ClickHouse, Kafka, Redis):

# Start minimal services
devservices up clickhouse kafka redis

# Or start all services including Snuba in containers
devservices up --mode=containerized

Services will be available at:

ClickHouse: localhost:9000 (native), localhost:8123 (HTTP)
Kafka: localhost:9093
Redis: localhost:6379

Initialize Database

Bootstrap Snuba and run migrations:

# Bootstrap Kafka topics and Redis
snuba bootstrap --force

# Run ClickHouse migrations
snuba migrations migrate --force

Start Snuba

Start the Snuba devserver:

snuba devserver

This starts:

API server on port 1218
Admin UI on port 1219
All consumers and background processes

Verify Installation

Test that Snuba is running correctly:

# Check API health
curl http://localhost:1218/health

# Should return: {"status": "ok"}

# Check ClickHouse connectivity
curl http://localhost:8123/ping

# Should return: Ok.

Method 2: Docker Compose

For containerized deployment, use the provided Docker Compose configuration.

Clone Repository

git clone https://github.com/getsentry/snuba.git
cd snuba

Review Docker Compose Config

The repository includes devservices/config.yml which defines all services:

clickhouse - ClickHouse database
snuba - Snuba API and consumers
Additional consumers for profiles, metrics, etc.

Services are configured with:

Automatic health checks
Proper networking
Volume mounts for data persistence

Start Services

Use devservices to start all containers:

devservices up --mode=containerized

Or manually with docker compose:

docker compose -f devservices/config.yml up -d

Initialize Database

Run migrations inside the container:

docker compose -f devservices/config.yml exec snuba \
  snuba migrations migrate --force

Docker Configuration

The Snuba container can be configured using environment variables:

environment:
  SNUBA_SETTINGS: docker
  DEBUG: 1
  CLICKHOUSE_HOST: clickhouse
  CLICKHOUSE_PORT: 9000
  CLICKHOUSE_HTTP_PORT: 8123
  DEFAULT_BROKERS: kafka:9093
  REDIS_HOST: redis
  REDIS_PORT: 6379
  REDIS_DB: 1
  
  # Enable optional consumers
  ENABLE_SENTRY_METRICS_DEV: 1
  ENABLE_PROFILES_CONSUMER: 1
  ENABLE_ISSUE_OCCURRENCE_CONSUMER: 1
  ENABLE_GROUP_ATTRIBUTES_CONSUMER: 1

Building Custom Docker Image

To build your own Snuba image:

# Build image
docker build -t snuba:custom .

# Run with custom image
SNUBA_IMAGE=snuba:custom docker compose up

Method 3: Local Development Setup

For active development on Snuba itself, install from source with all development tools.

Install System Dependencies

macOS
Linux (Ubuntu/Debian)
Linux (Fedora/RHEL)

# Install Homebrew packages
brew bundle

# This installs packages from Brewfile including:
# - Python 3.13
# - Rust
# - Other development tools

# Install Python 3.13
sudo apt update
sudo apt install python3.13 python3.13-dev python3-pip

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install build dependencies
sudo apt install build-essential libssl-dev pkg-config

# Install Python 3.13
sudo dnf install python3.13 python3.13-devel

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install build dependencies
sudo dnf install gcc gcc-c++ openssl-devel

Clone and Set Up Repository

git clone https://github.com/getsentry/snuba.git
cd snuba

# Use sentry-devenv if working on Sentry
# Or use make develop for standalone setup
make develop

This command:

Creates a virtual environment
Installs Python dependencies
Builds Rust components
Sets up pre-commit hooks

Install External Services

You still need ClickHouse, Kafka, and Redis. Use devservices:

devservices up clickhouse kafka redis

Or install them directly on your system:

macOS
Linux

brew install clickhouse kafka redis

# Start services
brew services start clickhouse
brew services start kafka
brew services start redis

Configure Snuba

Create a local configuration file:

# Copy example config
cp config/example.yml config/local.yml

# Edit configuration
vim config/local.yml

Minimal configuration:

clickhouse_host: localhost
clickhouse_port: 9000
clickhouse_http_port: 8123
brokers: localhost:9093
redis_host: localhost
redis_port: 6379

Initialize and Start

# Bootstrap
snuba bootstrap --force

# Run migrations
snuba migrations migrate --force

# Start devserver
snuba devserver

Development Tools

Snuba includes several make targets for development:

# Run tests
make test

# Run type checking
make type-check

# Build admin UI
make build-admin

# Watch and rebuild Rust on changes
make watch-rust-snuba

# Format code
make format-rust

# Generate config documentation
make generate-config-docs

Configuration

Settings Files

Snuba uses different settings profiles:

snuba.settings - Default settings
snuba.settings.docker - Docker/container settings
snuba.settings.test - Test environment settings

Set via environment variable:

export SNUBA_SETTINGS=docker

Key Configuration Options

ClickHouse Connection

# In settings.py or environment variables
CLICKHOUSE_HOST = "localhost"
CLICKHOUSE_PORT = 9000
CLICKHOUSE_HTTP_PORT = 8123
CLICKHOUSE_USER = "default"
CLICKHOUSE_PASSWORD = ""

Kafka Configuration

DEFAULT_BROKERS = ["localhost:9093"]
DEFAULT_KAFKA_CONSUMER_AUTO_OFFSET_RESET = "latest"

Redis Configuration

REDIS_HOST = "localhost"
REDIS_PORT = 6379
REDIS_DB = 1

API Server Settings

HOST = "0.0.0.0"
PORT = 1218
API_WORKERS = 4
API_THREADS = 8

Consumer Settings

# Enable/disable specific consumers
ENABLE_SENTRY_METRICS_DEV = True
ENABLE_PROFILES_CONSUMER = True
ENABLE_REPLAYS_CONSUMER = True
ENABLE_ISSUE_OCCURRENCE_CONSUMER = True

Migrations

Snuba uses a migration system to manage ClickHouse schema changes.

Running Migrations

snuba migrations migrate --force

Migration Groups

Migrations are organized by dataset/storage:

events - Events/errors storage
transactions - Transaction storage
metrics - Metrics storage
profiles - Profiling storage
replays - Replay storage
search_issues - Issue search storage

Always run migrations in a test environment first. Some migrations may be irreversible or require significant time to complete.

Verification

After installation, verify that everything is working:

Check Service Health

# Snuba API
curl http://localhost:1218/health

# ClickHouse
curl http://localhost:8123/ping

# Redis
redis-cli ping

Test Query Execution

curl -X POST http://localhost:1218/events/snql \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "MATCH (events) SELECT count() AS cnt WHERE timestamp >= toDateTime('"'2024-01-01T00:00:00'"') AND timestamp < toDateTime('"'2024-12-31T23:59:59'"') AND project_id = 1",
    "dataset": "events"
  }'

Check Admin UI

Open http://localhost:1219 in your browser to access the admin interface.

Verify Consumers

If running devserver, check that consumers are running:

# Should see multiple consumer processes
ps aux | grep "snuba rust-consumer"

Production Deployment

For production deployments, consider the following:

Scaling Considerations

API Servers

Run multiple API server instances behind a load balancer. Configure workers and threads based on CPU cores:

snuba api --processes=4 --threads=8

Consumer Replicas

Run multiple consumer instances for high throughput. Adjust batch timing based on replica count:

# With 12 replicas, use 12 second batches
--max-batch-time-ms=12000

ClickHouse Cluster

Use distributed ClickHouse setup with:

Multiple shards for horizontal scaling
Replication for high availability
Separate keeper/zookeeper cluster

Monitoring

Implement monitoring for:

Consumer lag
Query latency
ClickHouse performance
Error rates

Resource Requirements

Production resource recommendations:

Component	CPU	Memory	Disk
API Server	4+ cores	8GB+	20GB
Consumer	2+ cores	4GB+	20GB
ClickHouse	8+ cores	32GB+	SSD, size depends on data retention
Kafka	4+ cores	8GB+	500GB+
Redis	2+ cores	4GB+	10GB

Security

For production deployments:

Enable authentication on ClickHouse
Use TLS for all connections
Configure firewall rules
Set up proper access controls
Use secrets management for credentials

Troubleshooting

ClickHouse Connection Timeout

Symptoms: Connection timeout or Connection refused errorsSolutions:

Verify ClickHouse is running: curl http://localhost:8123/ping
Check ClickHouse logs: docker logs clickhouse
Ensure correct host/port in configuration
Check firewall rules

Kafka Consumer Lag

Symptoms: Slow data ingestion, growing consumer lagSolutions:

Increase consumer replicas
Reduce batch time
Check ClickHouse write performance
Monitor ClickHouse insert rate (should be ~1/sec per shard)

Migration Failures

Symptoms: Migration commands fail or hangSolutions:

Check ClickHouse connectivity
Review migration logs for errors
Ensure no other migrations are running
For hung migrations, check ClickHouse for long-running queries

Out of Memory Errors

Symptoms: Python or ClickHouse OOM errorsSolutions:

Increase memory allocation
Reduce batch sizes for consumers
Limit query complexity and row count
Configure ClickHouse memory limits

Rust Build Failures

Symptoms: make install-rs-dev failsSolutions:

Install Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Update Rust: rustup update
Install maturin: pip install maturin
Check for build dependencies

Next Steps

Now that you have Snuba installed:

Quickstart Guide

Learn how to execute your first queries and explore Snuba’s features

Configuration Guide

Deep dive into configuring datasets, entities, and query processors

API Reference

Complete reference for the HTTP API and query endpoints

Operations Guide

Best practices for monitoring, scaling, and maintaining Snuba

Get Started

Architecture

Query Languages

Datasets

Configuration

Migrations

Operations

Installation

Overview

Prerequisites

Method 1: Devservices (Recommended)

Verify Installation

Method 2: Docker Compose

Docker Configuration

Building Custom Docker Image

Method 3: Local Development Setup

Development Tools

Configuration

Settings Files

Key Configuration Options

Migrations

Running Migrations

Migration Groups

Verification

Production Deployment

Scaling Considerations

API Servers

Consumer Replicas

ClickHouse Cluster

Monitoring

Resource Requirements

Security

Troubleshooting

Next Steps

Quickstart Guide

Configuration Guide

API Reference

Operations Guide

Additional Resources

Build docs developers (and LLMs) love

Get Started

Architecture

Query Languages

Datasets

Configuration

Migrations

Operations

​Overview

​Prerequisites

​Method 1: Devservices (Recommended)

​Verify Installation

​Method 2: Docker Compose

​Docker Configuration

​Building Custom Docker Image

​Method 3: Local Development Setup

​Development Tools

​Configuration

​Settings Files

​Key Configuration Options

​Migrations

​Running Migrations

​Migration Groups

​Verification

​Production Deployment

​Scaling Considerations

API Servers

Consumer Replicas

ClickHouse Cluster

Monitoring

​Resource Requirements

​Security

​Troubleshooting

​Next Steps

Quickstart Guide

Configuration Guide

API Reference

Operations Guide

​Additional Resources

Build docs developers (and LLMs) love

Overview

Prerequisites

Method 1: Devservices (Recommended)

Verify Installation

Method 2: Docker Compose

Docker Configuration

Building Custom Docker Image

Method 3: Local Development Setup

Development Tools

Configuration

Settings Files

Key Configuration Options

Migrations

Running Migrations

Migration Groups

Verification

Production Deployment

Scaling Considerations

Resource Requirements

Security

Troubleshooting

Next Steps

Additional Resources