Skip to main content

​
Installation

Install LiteLLM with proxy dependencies: 
pip install 'litellm[proxy]'

​
Basic Setup

1

Create Configuration File

Create a config.yaml file with your model configurations:
config.yaml
model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: os.environ/OPENAI_API_KEY
  
  - model_name: claude-3-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: sk-1234  # Change this to a secure key
Never commit your master_key to version control. Use environment variables in production.
2

Set Environment Variables

Export your provider API keys:
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
3

Start the Proxy

Run the proxy server:
litellm --config config.yaml
The proxy will start on http://0.0.0.0:4000 by default.
litellm --config config.yaml --port 8000
4

Test the Proxy

Make a test request using curl:
curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ]
  }'

​
Generate a Virtual Key

Create an API key for your application: 
curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "models": ["gpt-3.5-turbo", "claude-3-sonnet"],
    "max_budget": 10.0,
    "duration": "30d"
  }'
Response: 
{
  "key": "sk-1234567890abcdef",
  "key_name": null,
  "expires": "2024-04-15T10:30:00Z",
  "models": ["gpt-3.5-turbo", "claude-3-sonnet"],
  "max_budget": 10.0
}

​
Use the Virtual Key

Now use the generated key instead of the master key: 
import openai

client = openai.OpenAI(
    api_key="sk-1234567890abcdef",
    base_url="http://localhost:4000"
)

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

​
Access the Admin UI

Open your browser to http://localhost:4000/ui to access the admin dashboard. Login with your master key (sk-1234) to:
  • View all virtual keys
  • Create and manage keys
  • Monitor usage and spending
  • View request logs
  • Manage teams and users

​
Load Balancing Example

Configure multiple deployments for automatic load balancing:
config.yaml
model_list:
  # OpenAI deployment 1
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY
      rpm: 480

  # OpenAI deployment 2 (backup)
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY_2
      rpm: 480

  # Azure fallback
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2024-02-15-preview"

router_settings:
  routing_strategy: usage-based-routing-v2
  enable_pre_call_checks: true

litellm_settings:
  num_retries: 3
  context_window_fallbacks: [{"gpt-4": ["claude-3-opus"]}]
Now requests to gpt-4 will automatically load balance across all three deployments, with fallback to Claude if all GPT-4 deployments fail.

​
Database Setup (Optional)

For persistent storage of keys, users, and spend data:
1

Set up PostgreSQL

# Using Docker
docker run -d \
  --name litellm-db \
  -e POSTGRES_DB=litellm \
  -e POSTGRES_USER=llmproxy \
  -e POSTGRES_PASSWORD=dbpassword9090 \
  -p 5432:5432 \
  postgres:16
2

Configure Database URL

export DATABASE_URL="postgresql://llmproxy:dbpassword9090@localhost:5432/litellm"
3

Enable DB Storage

Update your config.yaml:
general_settings:
  master_key: sk-1234
  store_model_in_db: true
  database_url: os.environ/DATABASE_URL
4

Start Proxy

The proxy will automatically run database migrations on startup:
litellm --config config.yaml

​
Health Check

Verify the proxy is running: 
curl http://localhost:4000/health
Response: 
{
  "status": "healthy",
  "db": "connected",
  "cache": "connected",
  "litellm_version": "1.x.x"
}

​
Next Steps

Docker Deployment

Deploy with Docker for production

Configuration Options

Explore all configuration settings

Virtual Keys

Advanced key management

Budget Alerts

Set up budget monitoring

​
Common Issues

​
Port Already in Use

If port 4000 is already in use, specify a different port: 
litellm --config config.yaml --port 8080

​
API Key Not Found

Make sure environment variables are exported: 
echo $OPENAI_API_KEY  # Should show your key

​
Database Connection Failed

Verify the DATABASE_URL format: 
postgresql://username:password@host:port/database

Build docs developers (and LLMs) love

Get started for freeTalk to us