Skip to main content

​
What is Parameter Golf?

OpenAI Model Craft Challenge: Parameter Golf is a competitive ML challenge to train the best language model that fits within a 16MB artifact and trains in under 10 minutes on 8xH100 GPUs, evaluated by compression on the FineWeb validation set (tokenizer-agnostic bits per byte). The challenge pushes participants toward creative solutions: novel architectures, aggressive compression schemes, and unconventional training strategies — all within a tight parameter budget.

Local Training

Start iterating on your Mac with Apple Silicon using the MLX training script

Remote GPU

Scale up to cloud GPUs on Runpod for full 8xH100 training runs

Scoring & Evaluation

Understand the bits-per-byte metric and how submissions are ranked

Submission Guide

Learn what you need to submit a valid SOTA record

​
Challenge Rules at a Glance

16MB Limit

Code bytes + compressed model bytes must total under 16,000,000 bytes (decimal, not MiB)

10-Minute Cap

Leaderboard submissions must train in under 10 minutes on 8xH100 SXM GPUs

BPB Metric

Scored on bits-per-byte compression of the FineWeb validation set — tokenizer-agnostic

​
Current Baseline

The naive baseline achieves 1.2244 BPB using a 9-layer, 512-dim transformer with a 1024-token vocabulary, grouped query attention (4 KV heads), and tied embeddings — all fitting within a 15.9MB compressed artifact.
OpenAI is sponsoring $1,000,000 in compute credits to help participants get started. Request a compute grant to access sponsored GPU time.

​
What Makes This Interesting

If you’re familiar with neural scaling laws, this challenge optimizes L(N) — the lowest loss achievable at a fixed parameter count N, unconstrained by data, compute, steps, or architecture. This is deliberately different from:
  • NanoGPT Speedrun — optimizes L(T): lowest time to reach a fixed loss target
  • NanoGPT Slowrun — optimizes L(D): lowest loss given a constrained dataset size
The 16MB constraint encourages exploration of approaches like:
  • Test-time compute and inference-time scaling
  • Aggressive parameter tying and depth recurrence
  • Low-rank training and quantization-aware training
  • Novel tokenizers and non-standard vocabularies
  • Megakernels and custom CUDA operations

​
Challenge Timeline

The challenge runs from March 18th to April 30th, 2026.

​
Community

Join the OpenAI Discord server and visit the Parameter Golf channels:
  • #parameter-golf-discussions — strategy and technique discussion
  • #parameter-golf-announcements — official updates

​
Careers at OpenAI

Many researchers at OpenAI first distinguished themselves through elite mathematics and programming competitions. The Model Craft Challenge is designed in that spirit — testing the ability to tackle unfamiliar problems with creativity and rigor.In June, OpenAI plans to hire a small cohort of early-career researchers. For exceptional participants, the challenge may serve as a way to stand out to OpenAI researchers and recruiters. Introduce yourself via the participant form.