This is only helpful for self-hosted users. If you’re using Khoj Cloud, you’re limited to our first-party models.
Khoj natively supports local LLMs available on HuggingFace in GGUF format. Using an OpenAI API proxy with Khoj may be useful for ease of setup, trying new models or using commercial LLMs via API.
Overview
LiteLLM exposes an OpenAI compatible API that proxies requests to other LLM API services. This provides a standardized API to interact with both open-source and commercial LLMs. Using LiteLLM with Khoj makes it possible to turn any LLM behind an API into your personal AI agent.Setup
Start LiteLLM Server
Start LiteLLM with your preferred model. For example, using Mistral tiny via Mistral API:
Create AI Model API
Create a new AI Model API on your Khoj admin panel:
- Name:
litellm - Api Key:
any string - Api Base Url:
<URL of your OpenAI Proxy API>
Create Chat Model
Create a new Chat Model on your Khoj admin panel:
- Name:
llama3.1(replace with the name of your local model) - Model Type:
Openai - Ai Model Api: the litellm AI Model API you created in step 3
- Max prompt size:
20000(replace with the max prompt size of your model) - Tokenizer: Do not set for OpenAI, Mistral, Llama3 based models
Select Model
Go to your config and select the model you just created in the chat model dropdown.
Supported Models
LiteLLM supports a wide range of LLM providers including:- OpenAI
- Anthropic
- Azure OpenAI
- Cohere
- Hugging Face
- Replicate
- And many more