The OgaModel class is the core interface for loading and managing generative AI models in ONNX Runtime GenAI. It provides methods to create model instances from configuration files or custom configurations.
Class Definition
struct OgaModel : OgaAbstract {
static std::unique_ptr<OgaModel> Create(const char* config_path);
static std::unique_ptr<OgaModel> Create(const char* config_path, const OgaRuntimeSettings& settings);
static std::unique_ptr<OgaModel> Create(const OgaConfig& config);
OgaString GetType() const;
OgaString GetDeviceType() const;
};
~/workspace/source/src/ort_genai.h:227
Methods
Create()
Create a new model instance from a configuration path.
static std::unique_ptr<OgaModel> Create(const char* config_path)
Path to the model configuration directory containing the model files and genai_config.json
std::unique_ptr<OgaModel> - A unique pointer to the created model instance
Throws: std::runtime_error if the model cannot be loaded
Example
// Load a model from a configuration path
auto model = OgaModel::Create("phi-2");
Create() with Runtime Settings
Create a model with custom runtime settings.
static std::unique_ptr<OgaModel> Create(const char* config_path, const OgaRuntimeSettings& settings)
Path to the model configuration directory
settings
const OgaRuntimeSettings&
required
Runtime settings for the model
std::unique_ptr<OgaModel> - A unique pointer to the created model instance
Example
// Create runtime settings
auto settings = OgaRuntimeSettings::Create();
settings->SetHandle("custom_handle", handle_ptr);
// Load model with settings
auto model = OgaModel::Create("phi-2", *settings);
Create() from Config
Create a model from a custom configuration object.
static std::unique_ptr<OgaModel> Create(const OgaConfig& config)
Custom configuration object for the model
std::unique_ptr<OgaModel> - A unique pointer to the created model instance
Example
// Create a custom config
auto config = OgaConfig::Create("phi-2");
config->AppendProvider("cuda");
config->SetProviderOption("cuda", "device_id", "0");
// Load model from config
auto model = OgaModel::Create(*config);
GetType()
Get the model type.
OgaString GetType() const
OgaString - The model type (e.g., “gpt2”, “llama”)
Example
auto model = OgaModel::Create("phi-2");
const char* model_type = model->GetType();
std::cout << "Model type: " << model_type << std::endl;
GetDeviceType()
Get the device type the model is running on.
OgaString GetDeviceType() const
OgaString - The device type (e.g., “cpu”, “cuda”, “dml”)
Example
auto model = OgaModel::Create("phi-2");
const char* device = model->GetDeviceType();
std::cout << "Running on: " << device << std::endl;
Complete Example
Here’s a complete example demonstrating model creation and basic usage:
#include "ort_genai.h"
#include <iostream>
int main() {
// Responsible for cleaning up the library during shutdown
OgaHandle handle;
try {
// Create model from configuration path
auto model = OgaModel::Create("phi-2");
// Get model information
std::cout << "Model type: " << model->GetType() << std::endl;
std::cout << "Device type: " << model->GetDeviceType() << std::endl;
// Create tokenizer and generator from the model
auto tokenizer = OgaTokenizer::Create(*model);
auto params = OgaGeneratorParams::Create(*model);
auto generator = OgaGenerator::Create(*model, *params);
// Use the model for generation...
} catch (const std::exception& e) {
std::cerr << "Error: " << e.what() << std::endl;
return -1;
}
return 0;
}
Resource Management
The OgaModel class uses RAII (Resource Acquisition Is Initialization) principles:
- Models are returned as
std::unique_ptr, providing automatic memory management
- The model is automatically destroyed when the unique pointer goes out of scope
- No manual cleanup is required
See Also