Skip to main content

​
Prerequisites

Before installing YC-Bench, ensure you have:
  • Python 3.12 or higher — YC-Bench uses modern Python features
  • uv — Fast Python package manager (automatically installed by the quickstart script)
  • API key for at least one LLM provider (Anthropic, OpenAI, Google, OpenRouter, etc.)
Python 3.11 and earlier are not supported. Use Python 3.12+ for compatibility.

​
Installation

1

Install uv (if not already installed)

YC-Bench uses uv for fast dependency management. Install it with:
curl -LsSf https://astral.sh/uv/install.sh | sh
After installation, add uv to your PATH:
export PATH="$HOME/.local/bin:$PATH"
Verify installation:
uv --version
2

Clone the repository

git clone https://github.com/collinear-ai/yc-bench.git
cd yc-bench
3

Install dependencies

uv sync
This command:
  • Creates a virtual environment
  • Installs all required dependencies from pyproject.toml
  • Sets up the yc-bench CLI tool
4

Verify installation

Test that the CLI is available:
uv run yc-bench --help
You should see the YC-Bench command help output.

​
API Key Setup

YC-Bench uses LiteLLM to support multiple LLM providers. Configure your API key by creating a .env file in the project root.

​
Supported Providers

# .env
ANTHROPIC_API_KEY="sk-ant-..."

# Example models:
# anthropic/claude-opus-4-6
# anthropic/claude-sonnet-4-6
# anthropic/claude-haiku-4-5-20251001
Multiple keys: You can include multiple provider keys in your .env file. YC-Bench will use the appropriate key based on the model you specify in the --model parameter.

​
Key Detection

The YC-Bench interactive launcher (yc-bench start) automatically detects your key type by prefix:
  • sk-ant-* → Anthropic
  • sk-or-* → OpenRouter
  • AIza* → Google Gemini
  • sk-* → OpenAI

​
Verification

Confirm your installation is working:
1

Check CLI availability

uv run yc-bench --help
Expected output:
Usage: yc-bench [OPTIONS] COMMAND [ARGS]...

Commands:
  run         Run a full benchmark
  start       Interactive 3-step quickstart
  company     Company status and info
  employee    Employee management
  market      Browse available tasks
  task        Task operations
  finance     Financial reports
  report      Monthly P&L reports
  sim         Simulation control
  scratchpad  Persistent memory
2

Test API key

Run a minimal test to verify your API key works:
uv run yc-bench start
The interactive wizard will:
  1. Detect your API key from .env or prompt for it
  2. Let you select a model
  3. Launch a test run
Press Ctrl+C to exit after confirming the connection works.

​
Troubleshooting

If uv is not found after installation:
  1. Ensure the install script completed successfully
  2. Add uv to your PATH: 
    export PATH="$HOME/.local/bin:$PATH"
  3. Restart your terminal or source your shell config: 
    source ~/.bashrc  # or ~/.zshrc
If you see requires-python = >=3.12 errors:
  1. Check your Python version: 
    python3 --version
  2. Install Python 3.12+ using your system package manager or pyenv
  3. Retry uv sync
If your API key isn’t detected:
  1. Verify the .env file is in the project root (same directory as pyproject.toml)
  2. Check for typos in the environment variable name
  3. Ensure no extra quotes or whitespace: 
    # Correct
ANTHROPIC_API_KEY="sk-ant-..."

# Incorrect
ANTHROPIC_API_KEY = "sk-ant-..."  # extra spaces
ANTHROPIC_API_KEY='sk-ant-...'    # single quotes (use double)
  4. Test manually: 
    source .env
echo $ANTHROPIC_API_KEY
If you encounter LiteLLM-related errors:
  1. Verify your API key is valid and has credits
  2. Check provider status pages for outages
  3. For OpenRouter, ensure your account has sufficient credits
  4. Try a different model from the same provider
  5. Check the LiteLLM documentation for provider-specific issues
YC-Bench uses SQLite by default. If you see database errors:
  1. Ensure the db/ directory exists (created automatically on first run)
  2. Check file permissions: ls -la db/
  3. Delete old database files if upgrading: rm db/*.db
  4. For PostgreSQL support: uv add yc-bench[postgres] and set DATABASE_URL in .env

​
Next Steps

Quickstart

Run your first benchmark with the interactive launcher

Build docs developers (and LLMs) love

Get started for freeTalk to us