Scaling llms.txt

With too few links in your /llms.txt file, agents miss pages that may contain relevant information. With too many, the agent is fed irrelevant information that bloats its context window, leading to more token spend and slower task completions.

When we added /llms.txt after its initial proposal in 2024, omitting a few links seemed like an acceptable tradeoff. Keep it simple, stupid (KISS). But over the past two years, as the docs sites we power have grown, the limits of the this tradeoff have started to be much more noticeable.

In need of a new solution

A June snapshot of one of our customer's /llms.txt contained 948 page links, but omitted 1,035. More pages were omitted than linked, and many of our larger customers had been experiencing similar situations.

To preserve coverage without expanding the root indefinitely, we set out to rebuild Mintlify's llms.txt generation architecture as a hierarchy of files.

As with our prior approach, we generate one file for small sites. But once that file exceeds the limit, /llms.txt remains the stable entry point and links to child files in the /_llms/ directory. Each child file contains a small set of links for a single subsection of the larger documentation navigation.

We ran a paired evaluation across ten large, public documentation sites using Claude Code with Sonnet 5 and Codex with GPT-5.6 Terra. With both agents weighted equally, site-balanced mean time to page fell 52.1% per task (363.6s to 174.2s), while mean input tokens fell 44.8% (1.74m to 960k).

One entry point, as many levels as the docs need

Under the new architecture, /llms.txt remains the root of the hierarchy. But once the root exceeds the limit, the generator moves complete navigation sections into child files.

/llms.txt
├── Guides → /_llms/guides.md
├── API reference → /_llms/api-reference.md
│   ├── Authentication → /_llms/api-reference/authentication.md
│   └── Endpoints → /_llms/api-reference/endpoints.md
└── Integrations → /_llms/integrations.md

Each child file contains either direct page links or a link to a deeper child, so an agent can follow the relevant branch and load only the files along that path. And as documentation teams add pages, the generator adds only as many hierarchy levels as necessary to represent the site's navigation within the size limit.

Grouping by navigation category

For those branches to guide retrieval, each must indicate what it contains. Splitting by character count would prevent truncation, but it would create ambiguous files: llms-2.txt and llms-3.txt give no indication of their contents.

Our new generator instead uses each site's navigation structure in docs.json, Mintlify's docs configuration file, to identify index groupings. Each product, version, language, group, or API reference section therefore becomes a candidate for its own child file. When an index grows too large, the generator chooses among those candidates by how much space each would save, while the navigation category tells agents what the resulting child file contains.

That navigation represents the publisher's product model by organizing pages around user intent, even when their URL paths follow a different structure. Child files therefore live under /_llms/*.md, where their paths can reflect the navigation hierarchy without implying equivalent site routes.

Move the branch that saves the most space

Once navigation defines the possible branches, the planner decides when and in what order to extract them. It first renders every section in the root file. If the result is 100,000 characters or fewer, it remains unchanged. When the result exceeds the limit, the planner selects the child section that produces the largest size reduction (the inline subtree size minus the child-file link size) and replaces that section with one child-file link.

render all sections inline

while the current file exceeds 100,000 characters:
    select the section with the largest size reduction
    replace the section with a child-file link

repeat these steps for each oversized child file

Throughout this process, the planner preserves navigation order and moves complete sections into child files before shortening descriptions or excluding pages. If one section contains thousands of direct pages, the planner uses the next URL path segment as a boundary. Only when no navigation or path boundary remains does it truncate the file and add an omission notice.

Evaluation

To compare the old and new llms.txt generation architectures, we froze snapshots of ten large Mintlify powered docs sites, on which we generated two arms, one for each of the llms.txt generation methods.

We then constructed a task set of 10 questions per docs site. Each task described a concrete need from a real target page without revealing its title or any unique URL segments. We constructed eight tasks for pages omitted from the old generation and two for visible pages, with the visible tasks serving as compatibility controls.

For each arm on each docs site, we ran Claude Code with Sonnet 5 and Codex with GPT-5.6 Terra. We ran three trials per task to account for the naturally noisy behavior of agents. Across those trials, we measured 404s hit per task, input tokens, and time to page, capped at the ten-minute per task limit.

Agents searched with fewer requests, tokens, and seconds

On tasks for pages omitted by the old architecture, we observed the following results:

Agent	404s hit per task	Input tokens	Time taken
Claude Sonnet 5	21.36 → 11.95 (−44.0%)	2.15M → 1.11M (−48.6%)	353.2s → 182.2s (−48.4%)
GPT-5.6 Terra	342.25 → 31.19 (−90.9%)	1.33M → 0.81M (−38.7%)	373.9s → 166.2s (−55.5%)
Average	181.80 → 21.57 (−88.1%)	1.74M → 0.96M (−44.8%)	363.6s → 174.2s (−52.1%)

Each model value is a site-balanced mean calculated from complete pairs. The average weights the two models equally.

Live for all Mintlify users

The new llms.txt generation architecture is now live for all Mintlify customers, auto-regenerated on each publish of your docs.

See the new llms.txt live in Mintlify's documentation.