Community Guides

​

Official Guides

​

Recent

​

Capabilities

​

Historical

​

Getting Started

​

Quick Start Paths

1. Boot an 8XH100 GPU node
2. Run bash runs/speedrun.sh (takes ~3 hours)
3. Chat with your model using python -m scripts.chat_web
4. See Leaderboard for optimization tips

1. Start with a d12 model (~5 minute training runs)
2. Try different depths: --depth=12, --depth=16, --depth=20
3. Monitor WandB for improvements in val_bpb and core_metric
4. See Contributing for best practices

1. Read the counting ‘r’ in strawberry guide
2. Create a new Task in tasks/ directory
3. Generate synthetic training data if needed
4. Add to SFT training mixture

​

Development Resources

​

Core Scripts

• runs/speedrun.sh - Current state-of-the-art GPT-2 training
• runs/scaling_laws.sh - Scaling law experiments
• runs/miniseries.sh - Train complete miniseries
• runs/runcpu.sh - Small example for CPU/Apple Silicon

​

Key Documentation

• dev/LOG.md - Development log with detailed explanations of changes
• dev/LEADERBOARD.md - Full leaderboard documentation
• README.md - Project overview and quick start

​

Example Commands

OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=12 \
    --run="my-experiment" \
    --model-tag="d12_experiment"

python -m scripts.base_eval \
    --checkpoint=checkpoints/d24_jan29.pt \
    --core-metric-max-per-task=100

# CLI interface
python -m scripts.chat_cli --checkpoint=checkpoints/d24_jan29.pt

# Web UI
python -m scripts.chat_web --checkpoint=checkpoints/d24_jan29.pt

​

Community Channels

​

Research Topics

​

Active Areas

• Training speed optimizations
• Better scaling laws
• Data curriculum strategies
• Mixed precision techniques

• Attention mechanism improvements
• Normalization strategies
• Initialization methods
• Parameter-efficient designs

• Additional benchmarks
• Faster evaluation methods
• Better metrics for small models

• SFT improvements
• RL training strategies
• Multi-task learning
• Synthetic data generation

​

Scaling Laws

• Model width (hidden dimension)
• Number of attention heads
• Learning rate schedules
• Training horizons (token:param ratio)
• Weight decay values
• Batch sizes (as of Run 3)

​

Tips and Tricks

​

Hardware

• ~$24/hr ($3/GPU/hr)
• GPT-2 in ~3 hours = ~$72
• Spot instances: ~$20 total

• Works fine, slightly slower
• More widely available

• Automatic gradient accumulation
• 8x longer training time
• Useful for debugging

• Reduce --device-batch-size to 16, 8, 4, 2, or 1
• Must be power of 2 for clean gradient accumulation

​

Training

# Reduce batch size (keeps total batch size via grad accum)
--device-batch-size=16  # or 8, 4, 2, 1

# Train smaller model for quick experiments
--depth=12              # ~5 min runs
--sample-every=-1       # Disable sampling
--save-every=-1         # Disable checkpointing
--core-metric-every=999999  # Only eval at end

# Hopper GPUs only (H100, H200)
--fp8                   # Faster but slightly lower quality
# Omit on Ampere (A100) - trains in bfloat16 automatically

​

Evaluation

--core-metric-max-per-task=100  # Faster, less accurate
--core-metric-max-per-task=-1   # Full eval (for leaderboard)

1. val_bpb - Validation loss (bits per byte)
2. core_metric - DCLM CORE score
3. train/mfu - Model FLOPs utilization
4. train/tok_per_sec - Training throughput

​

Related Projects

​

Citation

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that \$100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

​

Contributing Your Own Guide

1. Post in GitHub Discussions
2. Tag with appropriate labels (guide, tutorial, results, etc.)
3. Include:
   • Clear title and motivation
   • Step-by-step instructions
   • Code examples
   • Results and benchmarks
   • Links to resources