Skip to main content
The Tokenizer class handles text encoding and decoding for language models. It converts text into token sequences that models can process and decodes token sequences back into readable text.

​
Constructor

​
Tokenizer(Model model)

Creates a tokenizer instance from a loaded model.
​
model
Model
required
The model to create a tokenizer for
Tokenizer.cs:14-17
using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);

​
Encoding Methods

​
Encode(string str)

Encodes a single string into a token sequence.
​
str
string
required
The text to encode
Returns: Sequences - A sequences object containing the encoded tokens
Tokenizer.cs:74-87
string text = "Hello, how are you?";
using Sequences sequences = tokenizer.Encode(text);

​
EncodeBatch(string[] strings)

Encodes multiple strings into token sequences.
​
strings
string[]
required
Array of strings to encode
Returns: Sequences - A sequences object containing all encoded sequences
Tokenizer.cs:19-36
string[] texts = new string[] { 
    "First prompt",
    "Second prompt"
};
using Sequences sequences = tokenizer.EncodeBatch(texts);

​
Decoding Methods

​
Decode(ReadOnlySpan<int> sequence)

Decodes a token sequence into a string.
​
sequence
ReadOnlySpan<int>
required
The token sequence to decode
Returns: string - The decoded text
Tokenizer.cs:89-107
ReadOnlySpan<int> tokens = generator.GetSequence(0);
string decodedText = tokenizer.Decode(tokens);
Console.WriteLine(decodedText);

​
DecodeBatch(Sequences sequences)

Decodes multiple token sequences into strings.
​
sequences
Sequences
required
The sequences to decode
Returns: string[] - Array of decoded strings
Tokenizer.cs:38-47
string[] decodedTexts = tokenizer.DecodeBatch(sequences);
foreach (string text in decodedTexts)
{
    Console.WriteLine(text);
}

​
Chat Template Methods

​
ApplyChatTemplate(string template_str, string messages, string tools, bool add_generation_prompt)

Applies a chat template to format messages for the model.
​
template_str
string
required
The Jinja template string (can be empty to use model’s default)
​
messages
string
required
JSON-serialized array of chat messages
​
tools
string
required
JSON-serialized array of tools (can be empty)
​
add_generation_prompt
bool
required
Whether to add tokens indicating the start of the AI’s response
Returns: string - The formatted prompt
Tokenizer.cs:109-121
string messages = "[{\"role\":\"user\",\"content\":\"Hello!\"}]";
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);

​
Token ID Methods

​
GetBosTokenId()

Returns the beginning-of-sequence token ID. Returns: int - The BOS token ID
Tokenizer.cs:123-127
int bosTokenId = tokenizer.GetBosTokenId();

​
GetEosTokenIds()

Returns the end-of-sequence token IDs. Returns: ReadOnlySpan<int> - Span containing EOS token IDs
Tokenizer.cs:129-136
ReadOnlySpan<int> eosTokenIds = tokenizer.GetEosTokenIds();

​
GetPadTokenId()

Returns the padding token ID. Returns: int - The padding token ID
Tokenizer.cs:138-142
int padTokenId = tokenizer.GetPadTokenId();

​
Advanced Methods

​
UpdateOptions(Dictionary<string, string> options)

Updates tokenizer options.
​
options
Dictionary<string, string>
required
Dictionary of option names and values
Tokenizer.cs:49-72
var options = new Dictionary<string, string>
{
    { "add_special_tokens", "true" }
};
tokenizer.UpdateOptions(options);

​
CreateStream()

Creates a tokenizer stream for incremental decoding (useful for streaming generation). Returns: TokenizerStream - A tokenizer stream instance
Tokenizer.cs:144-149
using TokenizerStream tokenizerStream = tokenizer.CreateStream();

​
Complete Example

Here’s a complete example showing encoding, generation, and decoding:
examples/csharp/ModelChat/Program.cs:44-84
using Microsoft.ML.OnnxRuntimeGenAI;
using System.Text.Json;

string modelPath = "/path/to/model";

// Load model and create tokenizer
using Model model = new Model(modelPath);
using Tokenizer tokenizer = new Tokenizer(model);

// Prepare chat messages
string systemPrompt = "You are a helpful AI assistant.";
string userPrompt = "What is the capital of France?";

string messages = $"[{{\"role\":\"system\",\"content\":\"{systemPrompt}\"}}," +
                  $"{{\"role\":\"user\",\"content\":\"{userPrompt}\"}}]";

// Apply chat template and encode
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);

using var sequences = tokenizer.Encode(prompt);
Console.WriteLine($"Encoded {sequences.NumSequences} sequences");

// Create generator and generate response
using GeneratorParams generatorParams = new GeneratorParams(model);
using Generator generator = new Generator(model, generatorParams);

generator.AppendTokenSequences(sequences);

while (!generator.IsDone())
{
    generator.GenerateNextToken();
}

// Decode the output
var outputSequence = generator.GetSequence(0);
string outputString = tokenizer.Decode(outputSequence);

Console.WriteLine("Output:");
Console.WriteLine(outputString);

​
Streaming Example

examples/csharp/ModelChat/Program.cs:200-213
using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
using TokenizerStream tokenizerStream = tokenizer.CreateStream();

// ... setup generator and start generation ...

Console.Write("Output: ");
while (!generator.IsDone())
{
    generator.GenerateNextToken();
    // Decode and print each token as it's generated
    Console.Write(tokenizerStream.Decode(generator.GetNextTokens()[0]));
}
Console.WriteLine();

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us