What is LLM Gateway?
LLM Gateway sits between your application and LLM providers like OpenAI, Anthropic, Google, and AWS Bedrock. Instead of integrating with each provider separately, you make requests to the gateway using a unified OpenAI-compatible API, and it handles routing, authentication, caching, and logging.Key features
Unified API
OpenAI-compatible API that works with all major LLM providers
Multi-provider support
Connect to OpenAI, Anthropic, Google, AWS Bedrock, and more
Usage analytics
Track requests, tokens, costs, and performance metrics
Response caching
Reduce costs and latency with Redis-based caching
API key management
Generate and manage API keys with fine-grained permissions
Rate limiting
Control costs with token and request rate limits
Guardrails
Implement content filters and safety policies
MCP integration
Connect with Model Context Protocol-compatible tools
Architecture
LLM Gateway consists of two main services:Gateway API
The Gateway API provides OpenAI-compatible endpoints for making LLM requests:POST /v1/chat/completions- Chat completions with streaming supportPOST /v1/images/generations- Image generationGET /v1/models- List available modelsPOST /v1/messages- Anthropic-native message format
Management API
The Management API provides endpoints for managing your LLM Gateway instance:- API keys and provider keys
- Projects and organizations
- Usage logs and analytics
- Guardrails and content filters
- Billing and credits
Use cases
Unified interface
Replace provider-specific code with a single OpenAI-compatible API. Switch between providers without changing your application code.Cost optimization
Track spending across all providers, identify expensive requests, and implement guardrails to control costs.Multi-provider fallback
Configure automatic fallback to alternative providers when your primary provider is unavailable or rate-limited.Analytics and monitoring
Monitor token usage, response times, error rates, and costs across all your LLM requests in a unified dashboard.Compliance and safety
Implement guardrails to filter sensitive content, detect PII, and prevent jailbreak attempts before they reach the LLM provider.Deployment options
Hosted
The fastest way to get started is using the hosted version at llmgateway.io. Create an account, get an API key, and start making requests immediately.Self-hosted
For complete control over your data and infrastructure, you can self-host LLM Gateway using Docker or Kubernetes. See the self-hosting guide for detailed instructions.Getting started
Quickstart
Get up and running in 5 minutes
Self-hosting
Deploy on your own infrastructure
API reference
Explore the Gateway API
Integrations
Use with your favorite SDK or framework
Open source
LLM Gateway is open source and available on GitHub. The core functionality is licensed under AGPLv3, while enterprise features are available under a commercial license.Contributions are welcome! See the Contributing Guide to get started.
Next steps
Read the quickstart
Follow the quickstart guide to make your first API call in minutes.
Explore features
Learn about caching, analytics, and guardrails.
Integrate with your stack
See how to use LLM Gateway with OpenAI SDK, LangChain, or Vercel AI SDK.
Deploy to production
Review the self-hosting guide and enterprise features for production deployments.