This is only helpful for self-hosted users. If you’re using Khoj Cloud, you’re limited to our first-party models.
Khoj natively supports local LLMs available on HuggingFace in GGUF format. Using an OpenAI API proxy with Khoj may be useful for ease of setup, trying new models or using commercial LLMs via API.
Overview
LM Studio is a desktop app to chat with open-source LLMs on your local machine. LM Studio provides a neat interface for folks comfortable with a GUI. LM Studio can expose an OpenAI API compatible server. This makes it possible to turn chat models from LM Studio into your personal AI agents with Khoj.Setup (Deprecated)
Install LM Studio
Install LM Studio and download your preferred Chat Model.
Start Local Server
Go to the Server Tab on LM Studio, select your preferred Chat Model and click the green Start Server button.
Create AI Model API
Create a new AI Model API on your Khoj admin panel:
- Name:
lmstudio - Api Key:
any string - Api Base Url:
http://localhost:1234/v1/(default for LM Studio)
Create Chat Model
Create a new Chat Model on your Khoj admin panel:
- Name:
llama3.1(replace with the name of your local model) - Model Type:
Openai - Ai Model Api: the lmstudio AI Model API you created in step 3
- Max prompt size:
20000(replace with the max prompt size of your model) - Tokenizer: Do not set for OpenAI, Mistral, Llama3 based models
Select Model
Go to your config and select the model you just created in the chat model dropdown.
Alternatives
Since LM Studio integration is no longer supported, consider these alternatives:Ollama
Use Ollama to run local LLMs with full JSON mode support
LiteLLM
Proxy requests to various LLM providers through a unified API