Skip to main content
Parameter Golf uses a cached, retokenized version of the FineWeb dataset. The downloader fetches pre-tokenized binary shards and tokenizer models from a Hugging Face repository, so you don’t need to process raw text yourself.

​
Tokenizer variants

The baseline uses sp1024 — a 1024-token SentencePiece BPE vocabulary trained on FineWeb documents. The --variant flag selects which tokenizer family to download. Submissions that change the tokenizer will be examined carefully during review, since bugs may affect val_bpb scores.

​
Downloading data

The download script is data/cached_challenge_fineweb.py. Pass --variant and optionally --train-shards to control how much training data to fetch.
1

Download for local smoke testing (1 shard)

For quick local experiments, download a single training shard:
python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 1
2

Download the standard 8B token set (default)

The default download fetches the full validation split plus 80 training shards (8B tokens):
python3 data/cached_challenge_fineweb.py --variant sp1024
3

Download the full 10B token set

For the maximum training data available, fetch 100 shards:
python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 100
Each shard contains 100,000,000 tokens. 80 shards = 8B tokens; 100 shards = 10B tokens.

​
Directory layout after download

After running the downloader, your local layout looks like this: 
data/
  datasets/
    fineweb10B_sp1024/
      fineweb_train_*.bin   # Training shards
      fineweb_val_*.bin     # Validation shards (fixed set)
  tokenizers/
    fineweb_1024_bpe.model  # SentencePiece model
  manifest.json
  docs_selected.jsonl
  docs_selected.source_manifest.json

​
Validation split

The validation set is always the fixed first-50,000-document slice of FineWeb, stored in fineweb_val_* shards. It is always downloaded in full regardless of --train-shards. All val_bpb scores — local and leaderboard — are computed on this same split.

​
Using a custom dataset repository

If you have exported your own dataset to Hugging Face (for example, a 50B token export with a custom tokenizer), override the repo and path with environment variables: 
MATCHED_FINEWEB_REPO_ID=your-hf-username/your-dataset-repo \
MATCHED_FINEWEB_REMOTE_ROOT_PREFIX=your_50B_export_root \
python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 100
The default published repo is willdepueoai/parameter-golf, rooted under the datasets/ subdirectory. The downloader is manifest-driven, so it fetches only the prefix of shards you request from a larger export.

​
Rebuilding tokenizers

To retokenize from scratch, you first need the source documents. Pass --with-docs to the downloader to also fetch docs_selected.jsonl and its sidecar manifest: 
python3 data/cached_challenge_fineweb.py --variant sp1024 --with-docs
Then run the standalone retokenizer against the downloaded docs: 
python3 data/download_hf_docs_and_tokenize.py \
  --repo-id your-hf-username/your-dataset-repo \
  --remote-root your_50B_export_root \
  --output-root /tmp/my_custom_tokenizer_export \
  --tokenizer-config ./data/tokenizer_specs.json
The sidecar docs_selected.source_manifest.json includes a docs_sha256 field so you can verify you are rebuilding from the exact same document list and order as the baseline export.

​
Performance knobs

For CPU-heavy export jobs, the following environment variables control parallelism and batch sizes during shard tokenization:
VariableDescription
MATCHED_FINEWEB_SP_BATCH_SIZEBatch size for SentencePiece encoding (default: 2048)
MATCHED_FINEWEB_TOKENIZER_THREADSThread count for tokenizer encoding (default: 16)
MATCHED_FINEWEB_TIKTOKEN_THREADSThread count for tiktoken encoding (default: 16)
MATCHED_FINEWEB_GPT2_DECODE_BATCH_SIZEBatch size for GPT-2 decode in blobstore path (default: 512)
Example with custom parallelism: 
MATCHED_FINEWEB_SP_BATCH_SIZE=2048 \
MATCHED_FINEWEB_TOKENIZER_THREADS=16 \
MATCHED_FINEWEB_TIKTOKEN_THREADS=16 \
MATCHED_FINEWEB_GPT2_DECODE_BATCH_SIZE=512 \
python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 100

​
Next steps

Local training

Run your first training job on Apple Silicon with the downloaded data.

Remote GPU training

Scale up to cloud H100s for full leaderboard runs.