Quick Start - LeRobot

Overview

This guide will walk you through the essential workflows of LeRobot:

Loading and exploring datasets
Training a policy
Evaluating a trained policy
Running inference on real hardware

Make sure you have installed LeRobot before proceeding.

Working with Datasets

Loading a Dataset

LeRobot datasets are hosted on Hugging Face Hub and can be loaded with a single line:

from lerobot.datasets.lerobot_dataset import LeRobotDataset

# Load a dataset from the Hub
dataset = LeRobotDataset("lerobot/aloha_mobile_cabinet")

print(f"Number of episodes: {dataset.num_episodes}")
print(f"Number of frames: {dataset.num_frames}")

Exploring Dataset Metadata

Before downloading the full dataset, you can explore its metadata:

from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata

# Load only metadata (lightweight)
ds_meta = LeRobotDatasetMetadata("lerobot/aloha_mobile_cabinet")

print(f"Total episodes: {ds_meta.total_episodes}")
print(f"Frames per second: {ds_meta.fps}")
print(f"Robot type: {ds_meta.robot_type}")
print(f"Camera keys: {ds_meta.camera_keys}")
print(f"Tasks: {ds_meta.tasks}")

Accessing Dataset Items

# Access a single frame
frame = dataset[0]
print(f"Action shape: {frame['action'].shape}")
print(f"State shape: {frame['observation.state'].shape}")

# Access camera images (if available)
if ds_meta.camera_keys:
    camera_key = ds_meta.camera_keys[0]
    print(f"Image shape: {frame[camera_key].shape}")  # (C, H, W) in PyTorch format

Loading Temporal Data

For robot learning, you often need sequences of past observations and future actions:

# Define temporal offsets
delta_timestamps = {
    # Load 4 images: 1s before, 0.5s before, 0.2s before, and current
    "observation.image": [-1.0, -0.5, -0.2, 0.0],
    # Load 6 state vectors
    "observation.state": [-1.5, -1.0, -0.5, -0.2, -0.1, 0.0],
    # Load 64 future actions for action chunking
    "action": [t / ds_meta.fps for t in range(64)],
}

dataset = LeRobotDataset(
    "lerobot/aloha_mobile_cabinet",
    delta_timestamps=delta_timestamps
)

frame = dataset[0]
print(f"Image temporal shape: {frame['observation.image'].shape}")  # (4, C, H, W)
print(f"State temporal shape: {frame['observation.state'].shape}")  # (6, state_dim)
print(f"Action temporal shape: {frame['action'].shape}")  # (64, action_dim)

Using with PyTorch DataLoader

import torch

dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
)

for batch in dataloader:
    print(f"Batch action shape: {batch['action'].shape}")  # (32, 64, action_dim)
    break

Streaming Large Datasets

For very large datasets, use streaming mode to avoid downloading everything:

from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset

# Stream the massive DROID dataset (26M frames, 4TB!)
dataset = StreamingLeRobotDataset(
    "lerobot/droid_1.0.1",
    delta_timestamps={"action": [0.0]},
    tolerance_s=1e-3
)

# Data is fetched on-demand during iteration
dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, num_workers=4)

Training Your First Policy

Using the Command-Line Interface

The simplest way to train a policy:

lerobot-train \
  --policy=act \
  --dataset.repo_id=lerobot/aloha_mobile_cabinet

This command:

Downloads the dataset
Configures the ACT policy with default settings
Trains the model with automatic logging to Weights & Biases
Saves checkpoints periodically

Run lerobot-train --help to see all available options and configurations.

Training Programmatically

For more control, train policies programmatically:

Import Dependencies

from pathlib import Path
import torch
from lerobot.configs.types import FeatureType
from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
from lerobot.datasets.utils import dataset_to_policy_features
from lerobot.policies.act.configuration_act import ACTConfig
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors

Configure Training

# Setup
output_dir = Path("outputs/train/my_act_policy")
output_dir.mkdir(parents=True, exist_ok=True)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Dataset configuration
dataset_id = "lerobot/aloha_mobile_cabinet"
dataset_metadata = LeRobotDatasetMetadata(dataset_id)

Create Policy

# Extract features for policy configuration
features = dataset_to_policy_features(dataset_metadata.features)
output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
input_features = {key: ft for key, ft in features.items() if key not in output_features}

# Initialize policy with configuration
cfg = ACTConfig(input_features=input_features, output_features=output_features)
policy = ACTPolicy(cfg)
policy.train()
policy.to(device)

# Create preprocessor and postprocessor
preprocessor, postprocessor = make_pre_post_processors(
    cfg, dataset_stats=dataset_metadata.stats
)

Prepare Dataset

# Configure temporal data loading for ACT
delta_timestamps = {
    "action": [i / dataset_metadata.fps for i in cfg.action_delta_indices],
}

# Add image features if present
delta_timestamps.update({
    k: [i / dataset_metadata.fps for i in cfg.observation_delta_indices]
    for k in cfg.image_features
})

# Load dataset
dataset = LeRobotDataset(dataset_id, delta_timestamps=delta_timestamps)

# Create dataloader
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-4)
dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=device.type != "cpu",
    drop_last=True,
)

Training Loop

# Training configuration
training_steps = 5000
log_freq = 100

# Training loop
step = 0
done = False
while not done:
    for batch in dataloader:
        # Preprocess batch
        batch = preprocessor(batch)
        
        # Forward pass
        loss, _ = policy.forward(batch)
        
        # Backward pass
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        
        # Logging
        if step % log_freq == 0:
            print(f"Step {step}: loss = {loss.item():.4f}")
        
        step += 1
        if step >= training_steps:
            done = True
            break

Save Model

# Save checkpoint locally
policy.save_pretrained(output_dir)
preprocessor.save_pretrained(output_dir)
postprocessor.save_pretrained(output_dir)

# Optionally push to Hub
# policy.push_to_hub("<your-username>/my_act_policy")
# preprocessor.push_to_hub("<your-username>/my_act_policy")
# postprocessor.push_to_hub("<your-username>/my_act_policy")

The complete training example is available in the repository at examples/training/train_policy.py.

Evaluating Policies

Evaluation in Simulation

Evaluate your trained policy in a simulation environment:

lerobot-eval \
  --policy.path=lerobot/pi0_libero_finetuned \
  --env.type=libero \
  --env.task=libero_object \
  --eval.n_episodes=10

Supported simulation environments:

ALOHA: Bimanual manipulation tasks
PushT: 2D pushing task
LIBERO: Benchmark for lifelong robot learning (Linux only)
MetaWorld: Meta-learning benchmark with 50 tasks

Evaluation Results

The evaluation script outputs:

Success rate across episodes
Average reward per episode
Episode lengths
Videos of policy rollouts (saved to output directory)

Running on Real Hardware

Basic Inference Loop

Deploy a trained policy on real hardware:

import torch
from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.utils import build_inference_frame, make_robot_action
from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig

# Load trained policy
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_id = "<your-username>/my_act_policy"
policy = ACTPolicy.from_pretrained(model_id)
policy.to(device)
policy.eval()

# Load dataset metadata for preprocessing
dataset_id = "lerobot/aloha_mobile_cabinet"
dataset_metadata = LeRobotDatasetMetadata(dataset_id)
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, dataset_stats=dataset_metadata.stats
)

# Configure robot and cameras
camera_config = {
    "side": OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=30),
    "top": OpenCVCameraConfig(index_or_path=1, width=640, height=480, fps=30),
}

robot_cfg = SO100FollowerConfig(
    port="/dev/ttyUSB0",
    id="my_robot",
    cameras=camera_config
)

robot = SO100Follower(robot_cfg)
robot.connect()

# Run inference loop
max_steps = 100
for step in range(max_steps):
    # Get observation from robot
    obs = robot.get_observation()
    
    # Build frame matching dataset format
    obs_frame = build_inference_frame(
        observation=obs,
        ds_features=dataset_metadata.features,
        device=device
    )
    
    # Preprocess observation
    obs_frame = preprocessor(obs_frame)
    
    # Get action from policy
    with torch.no_grad():
        action = policy.select_action(obs_frame)
    
    # Postprocess action
    action = postprocessor(action)
    
    # Convert to robot action format
    action = make_robot_action(action, dataset_metadata.features)
    
    # Send action to robot
    robot.send_action(action)

robot.disconnect()

Always test your policy in simulation before deploying on real hardware. Start with slow speeds and ensure emergency stops are accessible.

Hardware-Specific Setup

For hardware-specific configuration:

# Find camera indices
lerobot-find-cameras

# Find robot serial port
lerobot-find-port

# Calibrate robot motors
lerobot-calibrate --robot=so100

# Set up motors (if needed)
lerobot-setup-motors --robot=so100 --port=/dev/ttyUSB0

Data Collection

Collect your own demonstrations:

Set Up Teleoperation

# Teleoperate the robot to test control
lerobot-teleoperate \
  --robot=so100 \
  --robot.port=/dev/ttyUSB0 \
  --teleoperator=gamepad

Record Demonstrations

# Record demonstrations to a new dataset
lerobot-record \
  --robot=so100 \
  --robot.port=/dev/ttyUSB0 \
  --teleoperator=gamepad \
  --output_dir=data/my_task \
  --num_episodes=50

Replay and Verify

# Replay recorded episodes to verify quality
lerobot-replay \
  --dataset_dir=data/my_task \
  --episode_index=0

Push to Hub (Optional)

from lerobot.datasets.lerobot_dataset import LeRobotDataset

# Load local dataset
dataset = LeRobotDataset.from_disk("data/my_task")

# Push to Hugging Face Hub
dataset.push_to_hub("<your-username>/my_task")

Visualizing Datasets

Visualize datasets to understand their structure:

# Launch interactive visualization
lerobot-dataset-viz --dataset.repo_id=lerobot/aloha_mobile_cabinet

This opens a Rerun viewer showing:

Camera feeds synchronized across time
Robot state trajectories
Action commands
Episode boundaries

Example Workflows

Workflow 1: Train on Existing Dataset

# Train ACT on ALOHA dataset
lerobot-train \
  --policy=act \
  --dataset.repo_id=lerobot/aloha_mobile_cabinet \
  --training.num_epochs=100 \
  --training.batch_size=32

# Evaluate in simulation
lerobot-eval \
  --policy.path=outputs/train/act \
  --env.type=aloha \
  --eval.n_episodes=10

Workflow 2: Collect Data and Train

# 1. Collect demonstrations
lerobot-record --robot=so100 --output_dir=data/my_task --num_episodes=50

# 2. Visualize collected data
lerobot-dataset-viz --dataset_dir=data/my_task

# 3. Train policy on collected data
lerobot-train --policy=act --dataset.local_dir=data/my_task

# 4. Deploy on robot
# (Use Python script for hardware inference)

Workflow 3: Fine-tune Pretrained Model

# Fine-tune a pretrained policy
lerobot-train \
  --policy.pretrained_path=lerobot/pi0_fast \
  --dataset.repo_id=<your-username>/my_custom_data \
  --training.learning_rate=1e-5 \
  --training.num_epochs=20

Next Steps

Hardware Integration

Set up your robot hardware and cameras

Policy Documentation

Learn about available policies and their configurations

Dataset Tools

Advanced dataset manipulation and tools

Examples

Explore more examples in the GitHub repository

Common Patterns

Pattern: Load, Train, Save

import torch
from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.act.configuration_act import ACTConfig
from lerobot.policies.factory import make_pre_post_processors
from lerobot.datasets.utils import dataset_to_policy_features
from lerobot.configs.types import FeatureType

# Load dataset and metadata
dataset_id = "lerobot/pusht"
metadata = LeRobotDatasetMetadata(dataset_id)
features = dataset_to_policy_features(metadata.features)

# Configure policy
output_features = {k: ft for k, ft in features.items() if ft.type is FeatureType.ACTION}
input_features = {k: ft for k, ft in features.items() if k not in output_features}
cfg = ACTConfig(input_features=input_features, output_features=output_features)

# Initialize
policy = ACTPolicy(cfg)
preprocessor, postprocessor = make_pre_post_processors(cfg, dataset_stats=metadata.stats)

# Train (simplified)
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-4)
# ... training loop ...

# Save
policy.save_pretrained("outputs/my_policy")
preprocessor.save_pretrained("outputs/my_policy")
postprocessor.save_pretrained("outputs/my_policy")

Pattern: Load and Infer

import torch
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata

# Load pretrained policy
policy = ACTPolicy.from_pretrained("<username>/my_policy")
policy.eval()

# Load preprocessor/postprocessor
dataset_id = "lerobot/aloha_mobile_cabinet"
metadata = LeRobotDatasetMetadata(dataset_id)
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, dataset_stats=metadata.stats
)

# Run inference
with torch.no_grad():
    obs = preprocessor(observation)
    action = policy.select_action(obs)
    action = postprocessor(action)

Additional Resources

Tutorial Space: Free hands-on robot learning course
Example Scripts: GitHub examples directory
Discord: Join the community
Documentation: Full API reference

Explore the examples/ directory in the repository for complete, runnable examples covering ACT, Diffusion Policy, VQ-BeT, and more.

Get Started

Core Concepts

Tutorials

Datasets

Simulation

Inference

Advanced

​Overview

​Working with Datasets

​Loading a Dataset

​Exploring Dataset Metadata

​Accessing Dataset Items

​Loading Temporal Data

​Using with PyTorch DataLoader

​Streaming Large Datasets

​Training Your First Policy

​Using the Command-Line Interface

​Training Programmatically

​Evaluating Policies

​Evaluation in Simulation

​Evaluation Results

​Running on Real Hardware

​Basic Inference Loop

​Hardware-Specific Setup

​Data Collection

​Visualizing Datasets

​Example Workflows

​Workflow 1: Train on Existing Dataset

​Workflow 2: Collect Data and Train

​Workflow 3: Fine-tune Pretrained Model

​Next Steps

Hardware Integration

Policy Documentation

Dataset Tools

Examples

​Common Patterns

​Pattern: Load, Train, Save

​Pattern: Load and Infer

​Additional Resources

Build docs developers (and LLMs) love

Overview

Working with Datasets

Loading a Dataset

Exploring Dataset Metadata

Accessing Dataset Items

Loading Temporal Data

Using with PyTorch DataLoader

Streaming Large Datasets

Training Your First Policy

Using the Command-Line Interface

Training Programmatically

Evaluating Policies

Evaluation in Simulation

Evaluation Results

Running on Real Hardware

Basic Inference Loop

Hardware-Specific Setup

Data Collection

Visualizing Datasets

Example Workflows

Workflow 1: Train on Existing Dataset

Workflow 2: Collect Data and Train

Workflow 3: Fine-tune Pretrained Model

Next Steps

Common Patterns

Pattern: Load, Train, Save

Pattern: Load and Infer

Additional Resources