Skip to main content

​
LLMProps

Properties for initializing and configuring a Large Language Model (LLM) instance.
​
model
object
required
Model configuration object
​
preventLoad
boolean
Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.

​
LLMType

React hook for managing a Large Language Model (LLM) instance.
​
messageHistory
Message[]
History containing all messages in conversation. This field is updated after model responds to sendMessage.
​
response
string
State of the generated response. This field is updated with each token generated by the model.
​
token
string
The most recently generated token.
​
isReady
boolean
Indicates whether the model is ready.
​
isGenerating
boolean
Indicates whether the model is currently generating a response.
​
downloadProgress
number
Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval.
​
error
RnExecutorchError | null
Contains the error message if the model failed to load.
​
configure
(config: LLMConfig) => void
Configures chat and tool calling. See Configuring the model for details.Parameters:
  • configuration (LLMConfig) - Configuration object containing chatConfig, toolsConfig, and generationConfig.
​
getGeneratedTokenCount
() => number
Returns the number of tokens generated so far in the current generation.
​
generate
(messages: Message[], tools?: LLMTool[]) => Promise<string>
Runs model to complete chat passed in messages argument. It doesn’t manage conversation context.Parameters:
  • messages (Message[]) - Array of messages representing the chat history.
  • tools (LLMTool[], optional) - Optional array of tools that can be used during generation.
Returns: The generated tokens as string.
​
getTotalTokenCount
() => number
Returns the number of total tokens from the previous generation. This is a sum of prompt tokens and generated tokens.
​
getPromptTokenCount
() => number
Returns the number of prompt tokens in the last message.
​
sendMessage
(message: string) => Promise<string>
Function to add user message to conversation. After model responds, messageHistory will be updated with both user message and model response.Parameters:
  • message (string) - The message string to send.
Returns: The model’s response as a string.
​
deleteMessage
(index: number) => void
Deletes all messages starting with message on index position. After deletion messageHistory will be updated.Parameters:
  • index (number) - The index of the message to delete from history.
​
interrupt
() => void
Function to interrupt the current inference.

​
LLMConfig

Configuration object for initializing and customizing a Large Language Model (LLM) instance.
​
chatConfig
Partial<ChatConfig>
Object configuring chat management, contains following properties:
  • systemPrompt - Often used to tell the model what is its purpose, for example - “Be a helpful translator”.
  • initialMessageHistory - An array of Message objects that represent the conversation history. This can be used to provide initial context to the model.
  • contextStrategy - Defines a strategy for managing the conversation context window and message history.
​
toolsConfig
ToolsConfig
Object configuring options for enabling and managing tool use. It will only have effect if your model’s chat template support it. Contains following properties:
  • tools - List of objects defining tools.
  • executeToolCallback - Function that accepts ToolCall, executes tool and returns the string to model.
  • displayToolCalls - If set to true, JSON tool calls will be displayed in chat. If false, only answers will be displayed.
​
generationConfig
GenerationConfig
Object configuring generation settings.
  • outputTokenBatchSize - Soft upper limit on the number of tokens in each token batch (in certain cases there can be more tokens in given batch, i.e. when the batch would end with special emoji join character).
  • batchTimeInterval - Upper limit on the time interval between consecutive token batches.
  • temperature - Scales output logits by the inverse of temperature. Controls the randomness / creativity of text generation.
  • topp - Only samples from the smallest set of tokens whose cumulative probability exceeds topp.

​
MessageRole

Roles that a message sender can have. 
type MessageRole = 'user' | 'assistant' | 'system';

​
Message

Represents a message in the conversation.
​
role
MessageRole
required
Role of the message sender of type MessageRole.
​
content
string
required
Content of the message.

​
ToolCall

Represents a tool call made by the model.
​
toolName
string
required
The name of the tool being called.
​
arguments
Object
required
The arguments passed to the tool.

​
LLMTool

Represents a tool that can be used by the model. Usually tool is represented with dictionary (Object), but fields depend on the model. Unfortunately there’s no one standard so it’s hard to type it better. 
type LLMTool = Object;

​
ChatConfig

Object configuring chat management.
​
initialMessageHistory
Message[]
required
An array of Message objects that represent the conversation history. This can be used to provide initial context to the model.
​
systemPrompt
string
required
Often used to tell the model what is its purpose, for example - “Be a helpful translator”.
​
contextStrategy
ContextStrategy
required
Defines a strategy for managing the conversation context window and message history.

​
ToolsConfig

Object configuring options for enabling and managing tool use. It will only have effect if your model’s chat template support it.
​
tools
LLMTool[]
required
List of objects defining tools.
​
executeToolCallback
(call: ToolCall) => Promise<string | null>
required
Function that accepts ToolCall, executes tool and returns the string to model.
​
displayToolCalls
boolean
If set to true, JSON tool calls will be displayed in chat. If false, only answers will be displayed.

​
GenerationConfig

Object configuring generation settings.
​
temperature
number
Scales output logits by the inverse of temperature. Controls the randomness / creativity of text generation.
​
topp
number
Only samples from the smallest set of tokens whose cumulative probability exceeds topp.
​
outputTokenBatchSize
number
Soft upper limit on the number of tokens in each token batch (in certain cases there can be more tokens in given batch, i.e. when the batch would end with special emoji join character).
​
batchTimeInterval
number
Upper limit on the time interval between consecutive token batches.

​
ContextStrategy

Defines a strategy for managing the conversation context window and message history.
​
buildContext
function
required
Constructs the final array of messages to be sent to the model for the current inference step.Parameters:
  • systemPrompt (string) - The top-level instructions or persona assigned to the model.
  • history (Message[]) - The complete conversation history up to the current point.
  • maxContextLength (number) - The maximum number of tokens that the model can keep in the context.
  • getTokenCount ((messages: Message[]) => number) - A callback function provided by the LLM controller that calculates the exact number of tokens a specific array of messages will consume once formatted.
Returns: The optimized array of messages, ready to be processed by the model.

​
SPECIAL_TOKENS

Special tokens used in Large Language Models (LLMs). 
export const SPECIAL_TOKENS = {
  BOS_TOKEN: 'bos_token',
  EOS_TOKEN: 'eos_token',
  UNK_TOKEN: 'unk_token',
  SEP_TOKEN: 'sep_token',
  PAD_TOKEN: 'pad_token',
  CLS_TOKEN: 'cls_token',
  MASK_TOKEN: 'mask_token',
};

Build docs developers (and LLMs) love

Get started for freeTalk to us