Running the Prover Server

The prover server is a Go application that exposes an HTTP API for generating zero-knowledge proofs. It can run in standalone mode or with Redis queue support for distributed processing.

Installation

Prerequisites

Go 1.21 or later
32+ GB RAM (64GB recommended for production)
16+ CPU cores
100+ GB disk space for proving keys

Building from Source

cd prover/server
go build -o light-prover .

Quick Start

Local Development

Start the prover server with minimal configuration:

./light-prover start \
  --keys-dir ./proving-keys/ \
  --prover-address 0.0.0.0:3001 \
  --auto-download

This will:

Start the HTTP server on port 3001
Auto-download proving keys as needed
Enable lazy loading (keys loaded on first use)

The --auto-download flag enables automatic downloading of missing proving keys. Without this flag, you must manually download keys using the download command.

Production Setup

# Download all keys first (recommended)
./light-prover download \
  --run-mode forester \
  --keys-dir ./proving-keys/

# Start with Redis queue for distributed processing
./light-prover start \
  --keys-dir ./proving-keys/ \
  --prover-address 0.0.0.0:3001 \
  --metrics-address 0.0.0.0:9998 \
  --redis-url redis://localhost:6379 \
  --preload-keys forester

Configuration

Command Line Flags

--prover-address

string

default:"0.0.0.0:3001"

HTTP server listen address

--metrics-address

string

default:"0.0.0.0:9998"

Prometheus metrics server address

--keys-dir

string

default:"./proving-keys/"

Directory for storing proving key files

--redis-url

string

Redis connection URL for queue mode (e.g., redis://localhost:6379)

--preload-keys

string

default:"none"

Preload keys at startup: none, all, or a run mode (rpc, forester, full)

--circuit

string[]

Specific circuits to enable (e.g., update, append, address-append)

--queue-only

boolean

Run only queue workers (no HTTP server)

--server-only

boolean

Run only HTTP server (no queue workers)

--auto-download

boolean

Automatically download missing key files

Downloading Proving Keys

Proving keys are large pre-computed files needed for proof generation.

By Run Mode

Download keys for specific deployment scenarios:

# Minimal keys for local testing
./light-prover download --run-mode local-rpc

By Circuit Type

Download specific circuit keys:

./light-prover download \
  --circuit update \
  --circuit append \
  --circuit address-append

Verify Existing Keys

./light-prover download \
  --run-mode forester \
  --verify-only

Run Modes

Run modes determine which circuits are available:

Mode	Circuits	Use Case
`local-rpc`	Test circuits (small batch sizes)	Local development
`rpc`	Inclusion, non-inclusion, combined	RPC nodes serving stateless.js
`forester`	Batch append, update, address append	Forester tree maintenance
`forester-test`	Test batch circuits	Forester testing
`full`	All production circuits	Full nodes
`full-test`	All circuits including tests	Development

API Endpoints

POST /prove

Generate a zero-knowledge proof.

curl -X POST http://localhost:3001/prove \
  -H "Content-Type: application/json" \
  -d '{
    "circuit": "update",
    "treeHeight": 26,
    "batchSize": 10,
    "oldRoot": "...",
    "newRoot": "...",
    "leaves": [...],
    "merkleProofs": [...]
  }'

GET /prove/status

Check proof job status.

curl "http://localhost:3001/prove/status?job_id=550e8400-e29b-41d4-a716-446655440000"

{
  "status": "processing",
  "message": "Generating proof..."
}

GET /health

Health check endpoint.

curl http://localhost:3001/health

{
  "status": "healthy",
  "uptime": 3600,
  "loaded_keys": 12
}

Redis Queue Mode

Architecture

Redis queue mode enables horizontal scaling:

Multiple Workers: Run multiple prover instances processing the same queue
Circuit-Specific Workers: Each worker type processes specific circuit types
Job Persistence: Jobs survive server restarts
Result Caching: Completed proofs cached for client retrieval

Starting Queue Workers

./light-prover start \
  --redis-url redis://localhost:6379 \
  --keys-dir ./proving-keys/

Queue Cleanup

The prover automatically cleans up old data:

Stuck Jobs: Every 10 seconds, requeue jobs stuck in processing state
Old Requests: Every 10 minutes, remove proof requests older than 1 hour
Failed Jobs: Every 30 minutes, remove failed jobs older than 1 hour
Old Results: Every 30 minutes, remove results older than 1 hour

Metrics

Prometheus metrics exposed on --metrics-address:

Key Metrics

prover_proof_generation_duration_seconds - Time to generate proofs by circuit type
prover_proof_requests_total - Total proof requests by circuit and status
prover_active_jobs - Currently processing jobs
prover_queue_depth - Pending jobs in queue by circuit type
prover_key_load_duration_seconds - Time to load proving keys

Example Prometheus Scrape Config

scrape_configs:
  - job_name: 'light-prover'
    static_configs:
      - targets: ['localhost:9998']

Monitoring

Logs

The prover outputs structured logs:

./light-prover start

Health Checks

# Basic health check
curl http://localhost:3001/health

# Check if server is responsive
if curl -sf http://localhost:3001/health > /dev/null; then
  echo "Prover is healthy"
else
  echo "Prover is down"
fi

Troubleshooting

Out of Memory Errors

Symptom: Server crashes or proofs fail with OOM errorsSolutions:

Increase available RAM (minimum 32GB, 64GB recommended)
Reduce concurrent proof generation
Use smaller batch sizes during development
Close other memory-intensive applications

Missing Proving Keys

Symptom: Error: “proving key file not found”Solutions:

Download keys: ./light-prover download --run-mode forester
Enable auto-download: --auto-download flag
Check disk space (keys require 100+ GB)
Verify keys directory permissions

Slow Proof Generation

Symptom: Proofs taking much longer than expectedSolutions:

Ensure adequate CPU cores (16+ recommended)
Check system CPU throttling (thermal issues)
Reduce concurrent proof jobs
Use SSD storage for key files (not HDD)
Monitor system load with top or htop

Constraint Errors

Symptom: “constraint system error” in proof generationCauses:

Invalid circuit inputs (wrong tree height, incorrect proofs)
Mismatched Merkle proof length and tree height
Invalid public input hash
Circuit input validation failed

Solutions:

Verify input data matches circuit parameters
Check that Merkle proofs are correct
Ensure public input hash is calculated correctly
Review circuit documentation for input requirements

Next Steps

Circuit Details

Learn about circuit implementations and input formats

Forester Integration

Use the prover with the Forester service

On-Chain Programs

ZK Prover

Forester Service

Indexer

Installation

Prerequisites

Building from Source

Quick Start

Local Development

Production Setup

Configuration

Command Line Flags

Downloading Proving Keys

By Run Mode

By Circuit Type

Verify Existing Keys

Run Modes

API Endpoints

POST /prove

GET /prove/status

GET /health

Redis Queue Mode

Architecture

Starting Queue Workers

Queue Cleanup

Metrics

Key Metrics

Example Prometheus Scrape Config

Monitoring

Logs

Health Checks

Troubleshooting

Next Steps

Circuit Details

Forester Integration

Build docs developers (and LLMs) love

On-Chain Programs

ZK Prover

Forester Service

Indexer

​Installation

​Prerequisites

​Building from Source

​Quick Start

​Local Development

​Production Setup

​Configuration

​Command Line Flags

​Downloading Proving Keys

​By Run Mode

​By Circuit Type

​Verify Existing Keys

​Run Modes

​API Endpoints

​POST /prove

​GET /prove/status

​GET /health

​Redis Queue Mode

​Architecture

​Starting Queue Workers

​Queue Cleanup

​Metrics

​Key Metrics

​Example Prometheus Scrape Config

​Monitoring

​Logs

​Health Checks

​Troubleshooting

​Next Steps

Circuit Details

Forester Integration

Build docs developers (and LLMs) love

Installation

Prerequisites

Building from Source

Quick Start

Local Development

Production Setup

Configuration

Command Line Flags

Downloading Proving Keys

By Run Mode

By Circuit Type

Verify Existing Keys

Run Modes

API Endpoints

POST /prove

GET /prove/status

GET /health

Redis Queue Mode

Architecture

Starting Queue Workers

Queue Cleanup

Metrics

Key Metrics

Example Prometheus Scrape Config

Monitoring

Logs

Health Checks

Troubleshooting

Next Steps