Skip to main content

Robot Kit API

Complete API reference for the six core robot tools.

Tool Overview

ToolInputsOutputsHardwareLatency
driveaction, distance/angleconfirmationMotors100ms-10s
lookaction, prompt (optional)image descriptionCamera2-5s
listenduration (seconds)transcribed textMicrophone1-10s
speaktextconfirmationSpeaker1-5s
senseaction, directionsensor readingsLIDAR, PIR, ultrasonic100ms-1s
emoteexpression, sound (optional)confirmationLED matrix, buzzer100-500ms

drive

Omni-directional motor control with safety integration.

Parameters

{
  "action": "forward" | "backward" | "strafe_left" | "strafe_right" | "rotate_left" | "rotate_right" | "stop",
  "distance": 1.0,  // meters (for linear movement)
  "angle": 90,       // degrees (for rotation)
  "speed": 0.5       // 0.0-1.0, optional, defaults to config max_speed
}

Examples

{
  "action": "forward",
  "distance": 1.5
}

Response

{
  "success": true,
  "output": "Moved forward 1.5 meters"
}

Backends

The drive tool supports multiple motor control backends:
BackendDescriptionConfiguration
mockLogs commands, no hardwarebackend = "mock"
serialArduino/motor controllerbackend = "serial", serial_port = "/dev/ttyACM0"
ros2ROS2 nav stackbackend = "ros2", ros2_topic = "/cmd_vel"
gpioDirect GPIO controlbackend = "gpio" (requires peripheral-rpi feature)

Safety Integration

When wrapped with SafeDrive, the tool:
  1. Pre-movement check: Queries sensors for obstacles
  2. Real-time monitoring: Adjusts speed based on proximity
  3. Collision response: Reverses if bump sensor triggered
  4. Watchdog: Auto-stops after timeout
  5. E-stop: Immediate halt on hardware button press
Always use SafeDrive wrapper in production. Direct DriveTool bypasses safety checks.

look

Capture images and generate AI descriptions using vision models.

Parameters

{
  "action": "capture" | "describe" | "find",
  "prompt": "a red ball",  // optional, for "find" action
  "save_path": "/tmp/image.jpg"  // optional, defaults to temp file
}

Examples

{
  "action": "describe"
}

Response

{
  "success": true,
  "output": "I see a couch, a TV on the wall, and some toys on the floor. The room is well-lit with natural light from a window."
}

Vision Models

Supported models (via Ollama):
ModelSizeSpeedAccuracyRecommended For
moondream1.7BFastGoodPi 4/5, general use
llava7BSlowBetterPi 5 only, detailed scenes
bakllava7BSlowBetterAlternative to llava
Configure in robot.toml:
[camera]
vision_model = "moondream"
ollama_url = "http://localhost:11434"

listen

Transcribe speech to text using Whisper.cpp.

Parameters

{
  "duration": 5,  // seconds to record
  "language": "en"  // optional, defaults to "en"
}

Examples

{
  "duration": 5
}

Response

{
  "success": true,
  "output": "Hey Buddy, can you help me find my toy?"
}

Whisper Models

ModelSizeSpeedAccuracyRAM Usage
tiny75 MBFastestFair200 MB
base142 MBFastGood400 MB
small466 MBMediumBetter1 GB
medium1.5 GBSlowBest2.5 GB
Recommendation: base for Pi 4/5 Configure in robot.toml:
[audio]
whisper_model = "base"
whisper_path = "/usr/local/bin/whisper-cpp"

speak

Text-to-speech using Piper TTS.

Parameters

{
  "text": "Hello! I'm ready to play!",
  "voice": "en_US-lessac-medium"  // optional, defaults to config
}

Examples

{
  "text": "I found the ball under the couch!"
}

Response

{
  "success": true,
  "output": "Spoke: Hello! I'm ready to play!"
}

Piper Voices

Available voices:
VoiceGenderStyleSpeed
en_US-lessac-mediumMaleClearMedium
en_US-amy-mediumFemaleFriendlyMedium
en_US-ryan-highMaleEnergeticFast
en_GB-alan-mediumMaleBritishMedium
Download from Piper Voices

sense

Query sensors: LIDAR, motion (PIR), ultrasonic distance.

Parameters

{
  "action": "scan" | "motion" | "distance",
  "direction": "front" | "all",  // for scan action
  "max_range": 5.0  // meters, optional
}

Examples

{
  "action": "scan",
  "direction": "front"
}

Response

{
  "success": true,
  "output": "Front: clear (2.3m), Left: obstacle at 0.8m, Right: clear (3.1m)"
}

Sensor Types

SensorRangeAccuracyUse Case
RPLidar A112m, 360°±5cmObstacle avoidance, mapping
HC-SR044m, narrow±1cmFront distance check
PIR Motion7m, 120°BinaryDetect people/animals
Configure in robot.toml:
[sensors]
lidar_port = "/dev/ttyUSB0"
lidar_type = "rplidar"
motion_pins = [17, 27]
ultrasonic_pins = [23, 24]

emote

Display LED expressions and play sound effects.

Parameters

{
  "expression": "happy" | "sad" | "excited" | "thinking" | "sleepy" | "surprised",
  "sound": "beep" | "chime" | "buzz"  // optional
}

Examples

{
  "expression": "happy"
}

Response

{
  "success": true,
  "output": "Displayed: happy face"
}

LED Patterns

8x8 LED matrix patterns (WS2812B):
Happy:    Sad:      Excited:  Thinking:
  ● ●       ● ●       ● ●         ● ●
  ● ●       ● ●       ● ●         ○ ●
                                    
 ●●●●●     ●●●●●     ●●●●●       ●●●

Error Handling

All tools return a ToolResult:
pub struct ToolResult {
    pub success: bool,
    pub output: String,
    pub error: Option<String>,
}

Common Errors

{
  "success": false,
  "output": "",
  "error": "Obstacle detected at 0.2m, stopping"
}
{
  "success": false,
  "output": "",
  "error": "Camera device /dev/video0 not found"
}
{
  "success": false,
  "output": "",
  "error": "E-stop pressed, all movement disabled"
}

Usage Examples

Hide and Seek

use zeroclaw_robot_kit::*;

#[tokio::main]
async fn main() {
    let config = RobotConfig::default();
    let tools = create_tools(&config);
    
    // Game sequence
    speak("I'll count to 20. Go hide!");
    emote("excited");
    tokio::time::sleep(Duration::from_secs(20)).await;
    
    speak("Ready or not, here I come!");
    
    loop {
        let obstacles = sense("scan", "all");
        drive("forward", 1.0);
        let scene = look("find", "a child hiding");
        
        if scene.contains("child") {
            speak("Found you!");
            emote("happy", Some("chime"));
            break;
        }
    }
}

Voice-Controlled Movement

loop {
    speak("Where should I go?");
    let command = listen(5).await?;
    
    match command.as_str() {
        cmd if cmd.contains("forward") => drive("forward", 1.0),
        cmd if cmd.contains("back") => drive("backward", 0.5),
        cmd if cmd.contains("left") => drive("rotate_left", 90),
        cmd if cmd.contains("right") => drive("rotate_right", 90),
        cmd if cmd.contains("stop") => {
            drive("stop", 0.0);
            break;
        },
        _ => speak("I didn't understand that"),
    }
}

Autonomous Patrol

loop {
    // Check for obstacles
    let scan = sense("scan", "front");
    if scan.min_distance < 0.5 {
        speak("Obstacle ahead, turning");
        drive("rotate_right", 90);
        continue;
    }
    
    // Check for motion
    let motion = sense("motion");
    if motion.detected {
        speak("I see movement!");
        let scene = look("describe");
        speak(&format!("I see: {}", scene));
    }
    
    // Continue patrol
    drive("forward", 2.0);
}

Safety Monitor

The SafetyMonitor runs as an independent task:
use zeroclaw_robot_kit::{SafetyMonitor, SafeDrive, DriveTool};

let config = RobotConfig::default();

// Create safety monitor
let (safety, safety_rx) = SafetyMonitor::new(config.safety.clone());
let safety = Arc::new(safety);

// Start sensor feed
tokio::spawn(async move {
    loop {
        let reading = get_lidar_data().await;
        safety.update_sensor(reading).await;
    }
});

// Wrap drive with safety
let drive = Arc::new(DriveTool::new(config.clone()));
let safe_drive = SafeDrive::new(drive, safety.clone());

// Now all drive commands go through safety checks
safe_drive.execute(json!({"action": "forward", "distance": 2.0})).await;

Safety Events

pub enum SafetyEvent {
    ObstacleDetected { distance: f64, angle: i32 },
    BumpSensorTriggered { sensor: String },
    EStopPressed,
    SensorTimeout,
    SpeedLimited { from: f64, to: f64, reason: String },
    MovementBlocked { reason: String },
}

Troubleshooting

Drive not working

# Check backend
grep backend ~/.zeroclaw/robot.toml

# Serial: verify port
ls /dev/ttyACM*

# ROS2: check topic
ros2 topic list | grep cmd_vel

# Test in mock mode
sed -i 's/backend = ".*"/backend = "mock"/' ~/.zeroclaw/robot.toml

Camera not found

# List cameras
ls /dev/video*
ffmpeg -f v4l2 -list_formats all -i /dev/video0

# Check permissions
sudo usermod -aG video $USER

Ollama vision fails

# Check Ollama is running
curl http://localhost:11434/api/tags

# Test vision model
ollama run moondream "Describe this image: /tmp/test.jpg"

# Use smaller model
sed -i 's/vision_model = ".*"/vision_model = "moondream"/' ~/.zeroclaw/robot.toml

Safety prevents movement

# Check sensor readings
tail -f /tmp/zeroclaw_sensors.fifo

# Increase safety thresholds (testing only!)
sed -i 's/min_obstacle_distance = 0.3/min_obstacle_distance = 0.15/' ~/.zeroclaw/robot.toml

# Disable safety (dangerous!)
sed -i 's/predict_collisions = true/predict_collisions = false/' ~/.zeroclaw/robot.toml

Next Steps

Setup Guide

Complete hardware and software setup

Overview

Architecture and features

Examples

Full example programs

Source Code

View robot kit source

Build docs developers (and LLMs) love