LLM Gateway Core
A production-grade infrastructure component that abstracts multiple Large Language Model providers behind a single, unified API
Why LLM Gateway Core?
LLM Gateway Core provides reliable, cost-effective access to multiple LLM providers with intelligent routing, distributed caching, atomic rate limiting, and comprehensive observability. Built on FastAPI with Redis and Prometheus integration.Multi-Provider Support
Abstract Gemini, Ollama, and custom providers behind a single API
Intelligent Routing
Route requests based on model hints: online, local, fast, or secure
Distributed Caching
Redis-backed response caching to reduce latency and API costs
Rate Limiting
Token bucket algorithm with atomic Redis operations
Quick Start
Get started with LLM Gateway Core in minutes.Quickstart
Deploy with Docker Compose and make your first request
Architecture
Understand the system design and core components
API Reference
Explore the REST API endpoints and schemas
Key Features
Provider Abstraction
Seamlessly integrate cloud and local model providers
Observability
Full metrics with Prometheus and Grafana dashboards
Docker Deployment
Production-ready Docker Compose configuration
Configuration
Flexible environment-based configuration