PARC Training Loop

​

Overview

​

Stage 0: Setup Iteration

​

Key Functions

1. Generate configuration files for all 4 stages
2. Compute sampling weights for motion classes (e.g., balance climbing vs. running motions)
3. Pre-compute terrain data for augmentation during MDM training
4. Set up directory structure for outputs

​

Configuration Structure

output_dir: "path/to/iteration_output"
input_mdm_config_path: "configs/mdm.yaml"
input_kin_gen_config_path: "configs/kin_gen.yaml"
input_tracker_config_path: "configs/tracker.yaml"
input_phys_record_config_path: "configs/phys_record.yaml"
iter_start_dataset_path: "data/motions.yaml"

​

Stage 1: Train Generator

​

Process Flow

1. Load or create motion sampler from dataset
   • Motion sampler handles weighted sampling with replacement
   • Pre-computes heightfield data for each motion
   • Caches sampler to disk for faster subsequent loads
2. Compute dataset statistics
   • Mean and std for motion features across all sequences
   • Used for feature normalization during training
   • Cached to sampler_stats.txt
3. Initialize or load MDM model
   • Can start from scratch or fine-tune existing checkpoint
   • Supports EMA (Exponential Moving Average) for stable training
4. Training loop
   • Random timestep sampling (0 to diffusion_timesteps)
   • Forward diffusion: add noise to clean motions
   • Denoising: predict clean motion from noisy input
   • Multiple loss terms (see Motion Diffusion page)

​

Training Configuration

epochs: 1000
batch_size: 512
iters_per_epoch: 100
diffusion_timesteps: 1000
learning_rate: 0.0001
weight_decay: 0.0

​

Outputs

• checkpoints/model_*.ckpt - Periodic checkpoints
• final_model.ckpt - Final trained model
• sampler.pkl - Cached motion sampler
• sampler_stats.txt - Dataset statistics

​

Stage 2: Kinematic Generation

​

Generation Pipeline

​

Detailed Steps

​

1. Terrain Generation

• Random boxes - scattered box obstacles
• Random stairs - ascending/descending staircases
• Random paths - narrow walkways

​

2. Path Planning

• Nodes: grid cells in heightfield
• Edges: traversable connections (consider height changes)
• Cost: distance + height penalty

​

3. Autoregressive Generation

• Start with initial pose
• Generate sequence segment conditioned on heightmap + target direction
• Use end of previous segment as start of next
• Repeat until path is complete

​

4. Candidate Selection

• Foot contact consistency
• Smooth velocity profiles
• Minimal ground penetration
• Path following accuracy

​

5. Hesitation Removal

• Low velocity + low acceleration = hesitation
• Remove consecutive hesitation frames
• Re-blend motion segments

​

6. Kinematic Optimization

• Foot contact constraints (IK)
• Body position targets
• Smooth velocity constraints
• Collision avoidance

​

Batch Generation

kin_gen_num_batches_of_motions: 10
kin_gen_num_motions_per_batch: 50
kin_gen_motion_id_offset: 0

​

Outputs

• ignore/raw/ - Unoptimized generated motions
• <batch_name>/ - Optimized motions ready for tracking
• Motion files in .ms format with terrain data

​

Stage 3: Train Tracker

​

Environment Setup

• Simulated character - follows PD controller to track reference
• Reference character - plays back kinematic motion
• Terrain - heightfield collision geometry
• Parallel envs - thousands of environments in parallel (one per motion)

​

Observation Space

• Character state (joint positions, velocities, root state)
• Reference motion (target poses from kinematic motion)
• Heightmap (local terrain geometry)
• Contact labels (target foot contacts)

​

Reward Function

• Root position error - distance between sim and ref root
• Root rotation error - angular difference
• Body position error - per-body position differences
• Body rotation error - per-body angular differences
• Joint velocity error - velocity matching
• End effector error - foot/hand position matching

​

Training Details

• Algorithm: PPO with GAE (Generalized Advantage Estimation)
• Policy network: MLP with 2-4 hidden layers (512-1024 units)
• Value network: Shared or separate MLP
• Updates: Multiple epochs per rollout buffer
• Clipping: PPO ratio clipping (ε = 0.2)

​

Adaptive Motion Weighting

• Easier motions (low fail rate) get lower weight
• Harder motions (high fail rate) get higher weight
• Fail rates updated via exponential moving average

self._ema_weight = 0.01
fail_rate = success_rate.new()
new_fail_rate = (1 - ema_weight) * old_fail_rate + ema_weight * fail_rate

​

Grid Layout

[Env 0] [Env 1] [Env 2] ...
[Env N] [Env N+1] ...
...

​

Outputs

• model.pt - Trained PPO policy
• agent_config.yaml - Agent configuration
• dm_env.yaml - Environment configuration
• fail_rates_*.pt - Per-motion failure rates

​

Stage 4: Physics Recording

​

Recording Process

1. Load trained controller from Stage 3
2. Load generated motions from Stage 2
3. Simulate in parallel - run controller on all motions simultaneously
4. Record successful tracks - save motions that complete without early termination
5. Retry with offsets - for partial failures, try starting from later frames

​

Retry Strategy

1. Try starting from beginning (0% into motion)
2. If fails, try starting from 25% into motion
3. If fails, try starting from 50% into motion
4. If all fail, discard motion

​

Quality Control

• Complete without early termination
• Maintain low tracking error throughout
• Stay within terrain bounds
• Don’t violate physics constraints

​

Outputs

• Physically valid motion files (.ms format)
• Only successfully tracked motions included
• These motions are added to the dataset for next iteration

​

Iteration Management

# Setup iteration
python scripts/parc_0_setup_iter.py --config configs/setup.yaml

# Stage 1: Train generator
python scripts/parc_1_train_gen.py --config output/p1_train_gen/mdm_config.yaml

# Stage 2: Generate motions (run in parallel)
for config in output/p2_kin_gen/*/kin_gen_config.yaml; do
    python scripts/parc_2_kin_gen.py --config $config
done

# Stage 3: Train tracker  
python scripts/parc_3_tracker.py --config output/p3_tracker/tracker.yaml

# Stage 4: Record physics
python scripts/parc_4_phys_record.py --config output/p4_phys_record/phys_record.yaml

​

Next Iteration

1. Update iter_start_dataset_path to include physically recorded motions from iteration N
2. Optionally use final MDM checkpoint as input_mdm_model_path
3. Optionally use final tracker checkpoint as input_tracker_model_path
4. Run parc_0_setup_iter.py with updated config
5. Execute 4 stages again

​

Performance Considerations

• GPU memory: ~8-16GB for batch size 512
• Training time: Hours to days depending on epochs
• Checkpoint frequently for long runs

• CPU intensive for optimization
• Highly parallelizable (run batches independently)
• Can take hours per batch with optimization

• Requires GPU with Isaac Gym support
• Most computationally expensive stage
• Training time: Days for complex motions
• Scales with number of motions × terrains per motion

• Fast (minutes to hours)
• Parallel execution across all environments
• GPU memory scales with number of motions