Skip to main content

Overview

This example demonstrates how to create an LLM-powered chat agent with streaming capabilities. You’ll learn how to:
  • Create an LLM agent using OpenAIProvider
  • Perform simple Q&A without context retention
  • Build multi-turn conversations with context
  • Stream responses in real-time
  • Get both streaming and full responses

What You’ll Learn

  • Setting up OpenAI provider from environment
  • Using LLMAgentBuilder for agent configuration
  • Difference between ask() and chat() methods
  • Implementing streaming with ask_stream() and chat_stream()
  • Handling streaming responses with tokio-stream

Prerequisites

  • Rust 1.75 or higher
  • OpenAI API key (set as OPENAI_API_KEY environment variable)
  • Optional: Custom API endpoint (e.g., Ollama) via OPENAI_BASE_URL

Complete Source Code

View the complete example from the MoFA repository:
use mofa_sdk::llm::{LLMAgentBuilder, OpenAIProvider};
use tokio_stream::StreamExt;
use tracing::{info, Level};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt()
        .with_max_level(Level::INFO)
        .init();
    info!("========================================");
    info!("  MoFA LLM Agent                       ");
    info!("========================================\n");

    // Create OpenAI provider from environment variables
    let openai_provider = OpenAIProvider::from_env();
    
    // Build agent with OpenAI provider
    let agent = LLMAgentBuilder::new()
        .with_provider(std::sync::Arc::new(openai_provider))
        .build();
    
    info!("Agent loaded: {}", agent.config().name);
    info!("Agent ID: {}\n", agent.config().agent_id);

    // Demo: Interactive chat
    info!("--- Chat Demo ---\n");

    // Simple Q&A (no context retention)
    let response = agent
        .ask("Hello! What can you help me with?")
        .await
        .map_err(|e| -> Box<dyn std::error::Error> { 
            format!("LLM error: {e}").into() 
        })?;
    info!("Q: Hello! What can you help me with?");
    info!("A: {response}\n");

    // Multi-turn conversation (with context retention)
    info!("--- Multi-turn Conversation ---\n");

    let r1 = agent
        .chat("My favorite programming language is Rust.")
        .await
        .map_err(|e| -> Box<dyn std::error::Error> { 
            format!("LLM error: {e}").into() 
        })?;
    info!("User: My favorite programming language is Rust.");
    info!("AI: {r1}\n");

    let r2 = agent
        .chat("What's my favorite language?")
        .await
        .map_err(|e| -> Box<dyn std::error::Error> { 
            format!("LLM error: {e}").into() 
        })?;
    info!("User: What's my favorite language?");
    info!("AI: {r2}\n");

    // Streaming Q&A
    info!("--- Streaming Q&A ---\n");
    let mut stream = agent.ask_stream("Tell me a story").await?;
    while let Some(result) = stream.next().await {
        match result {
            Ok(text) => print!("{text}"),
            Err(e) => info!("Error: {e}"),
        }
    }
    println!("\n");

    // Streaming multi-turn conversation
    info!("--- Streaming Chat ---\n");
    let mut stream = agent.chat_stream("Hello!").await?;
    while let Some(result) = stream.next().await {
        if let Ok(text) = result {
            print!("{text}");
        }
    }
    println!("\n");

    // Streaming with full response
    info!("--- Streaming with Full Response ---\n");
    let (mut stream, full_rx) = agent
        .chat_stream_with_full("What's 2+2?")
        .await?;
    
    while let Some(result) = stream.next().await {
        if let Ok(text) = result {
            print!("{text}");
        }
    }
    
    let full_response = full_rx.await?;
    info!("\nFull: {full_response}");
    
    info!("========================================");
    info!("  Demo completed!                      ");
    info!("========================================");

    Ok(())
}

Running the Example

1
Set Environment Variables
2
export OPENAI_API_KEY="your-api-key-here"

# Optional: Use custom endpoint (e.g., Ollama)
export OPENAI_BASE_URL="http://localhost:11434/v1"
3
Run the Example
4
cd examples/chat_stream
cargo run

Expected Output

========================================
  MoFA LLM Agent
========================================

Agent loaded: LLM Agent
Agent ID: llm_agent_001

--- Chat Demo ---

Q: Hello! What can you help me with?
A: Hello! I'm an AI assistant powered by OpenAI. I can help you with...

--- Multi-turn Conversation ---

User: My favorite programming language is Rust.
AI: That's great! Rust is known for its memory safety...

User: What's my favorite language?
AI: Based on what you just told me, your favorite programming language is Rust!

--- Streaming Q&A ---

Once upon a time...
...

========================================
  Demo completed!
========================================

Key Concepts

ask() vs chat()

The ask() method performs stateless queries. Each call is independent with no context retention.
let response = agent.ask("What is Rust?").await?;
// Next ask() won't remember this conversation
The chat() method maintains conversation context across multiple turns.
agent.chat("My name is Alice").await?;
let response = agent.chat("What's my name?").await?;
// Response: "Your name is Alice"

Streaming Responses

MoFA provides multiple streaming methods:
Stream responses for stateless queries:
let mut stream = agent.ask_stream("Tell me a story").await?;
while let Some(result) = stream.next().await {
    match result {
        Ok(text) => print!("{text}"),
        Err(e) => eprintln!("Error: {e}"),
    }
}

Configuration Options

Environment Variables

VariableRequiredDefaultDescription
OPENAI_API_KEYYes-Your OpenAI API key
OPENAI_BASE_URLNohttps://api.openai.com/v1Custom API endpoint
OPENAI_MODELNogpt-4Model to use

Builder Options

let agent = LLMAgentBuilder::new()
    .with_provider(Arc::new(provider))
    .with_name("My Agent")
    .with_system_prompt("You are a helpful assistant")
    .with_temperature(0.7)
    .with_max_tokens(2048)
    .build();

Common Use Cases

Customer Support

Build chatbots with context-aware responses

Content Generation

Stream generated content in real-time

Code Assistant

Help users with programming questions

Data Analysis

Query and analyze data conversationally

Troubleshooting

Error: OPENAI_API_KEY not foundSolution: Set your API key:
export OPENAI_API_KEY="sk-..."
Error: Connection timeoutSolution: Check your network or use a proxy:
export HTTPS_PROXY="http://proxy.example.com:8080"
Error: Rate limit exceededSolution: Implement retry logic or upgrade your API plan

Next Steps

ReAct Agent

Add reasoning and tool use

Multi-Agent

Coordinate multiple agents

LLM Integration

Deep dive into LLM features

Streaming Guide

Advanced streaming patterns

Build docs developers (and LLMs) love