---
title: "What an Agent Reads Before It Recommends You"
description: "The docs, markdown pages, and llms.txt entries an agent opens before naming a product, and how those files connect to a recommendation or a finished task."
canonical: "https://www.agentrail.co/blog/documentation-for-ai-agents"
author: "Vivek Mittal"
published: "2026-11-03T00:00:00Z"
last_updated: "2026-11-03T11:27:13.779Z"
---

# What an Agent Reads Before It Recommends You

The docs, markdown pages, and llms.txt entries an agent opens before naming a product, and how those files connect to a recommendation or a finished task.

![Orange pixel robot connected to a highlighted card labeled Your product, beside two dimmed options](https://images.ctfassets.net/1htfyfnn1o5y/7uoRP2rncQgviIZz2v35Md/df433ad2f1c266f8a3f608ccf0535255/6d0e42fd-230d-45e1-9389-b8ceca2253ec.png?w=1600&fm=webp&q=72)

Before an agent names your product, it opens a short set of files. Those files are the documentation for AI agents that actually matter: a curated index, a handful of markdown pages, and sometimes a machine-readable API contract. The nav your humans love rarely appears in that path.

This page is about that reading path. It connects the files an agent opens to a [coding agent recommendation](https://www.agentrail.co/blog/coding-agent-recommendations) or a finished task. [Agent discoverability](https://www.agentrail.co/blog/agent-discoverability) asks whether the agent can find the doorway. [Presence versus use](https://www.agentrail.co/blog/llms-txt-for-ai-agents) asks whether a published file was fetched. Here the job is narrower: once the agent is looking, what should it read, and how do you prove those reads steered the outcome?

> **Key Takeaways**
>
> • Agents fetch an index, then a few LLM-friendly pages. They do not walk your human docs tree.
>
> • [`llms.txt`](https://llmstxt.org/) is the map. Markdown pages are the evidence the agent can quote, install from, or call against.
>
> • A map that overflows the context window fails before a useful page loads. Cloudflare’s split-index work is public proof of that pattern.
>
> • Shipping a file is not the same as an agent opening it. That presence-versus-use split is covered separately.
>
> • Audit the URLs opened before a product name or a stop. That trail is how you connect documentation to preference or completion.

## The recommendation path starts with a short reading list

An agent does not browse. It fetches. Given a job like “add email sending” or “wire OAuth,” it looks for a short overview of what exists, then follows a small number of links into detail. [Agent experience](https://www.agentrail.co/blog/agent-experience) is that encounter end to end. Documentation for AI agents is the subset of surfaces that get opened while the agent is still deciding what to name, install, or call.

Jeremy Howard’s [`llms.txt` proposal](https://llmstxt.org/), first published 3 September 2024 and updated 10 August 2026, states the expected loop plainly. Agents view or search `llms.txt`, then follow the relevant links. Those links should point at LLM-friendly content, usually markdown, not at another HTML shell. The index stays small enough for a context window. Detail lives behind the links and is fetched only when needed.

That is a different success criterion from human documentation. Humans tolerate chrome, sidebars, and “related articles.” Agents pay in tokens for every wasted fetch. A page that only restates a child list costs a round trip and still leaves the agent without the fact it needed.

## llms.txt is the map; markdown pages are the evidence

Treat `llms.txt` as the map and the linked pages as the evidence. The map steers which evidence loads. The evidence supplies the claim the agent can act on: an install command, a current endpoint, a header name, a warning that names a failing approach.

The format is markdown at `/llms.txt`, or at a path such as `/docs/llms.txt`. A file covers the URLs under its path. The most specific file wins when more than one applies. The only required section is an H1 with the name of the site or project. A blockquote can summarize it. H2 sections hold lists of links. Each item is a markdown link, optionally followed by a colon and a short note. Those notes are steering. Vague labels force the agent to open more pages to learn what each link contains.

Howard also proposes a markdown twin of pages that matter, at the same URL with `.md` appended or with the extension replaced, advertised with `rel="alternate"` and `type="text/markdown"`. Content negotiation (`Accept: text/markdown`) helps when the agent sends that header. A stable `/index.md`-style URL helps when it does not. Either way, the fact the agent needs must exist in bytes the fetch returns without running your client JavaScript.

OpenAPI belongs in the same path when the job is an HTTP call. Prose can describe an API. A linked contract reduces the chance the agent invents a path that never existed. Discoverability of that contract is covered in [agent discoverability](https://www.agentrail.co/blog/agent-discoverability). This page only insists that if the recommendation or the task depends on a call, the reading list should point at the contract the agent is supposed to use.

## A map that does not fit the context window fails before a page is read

A single `llms.txt` that lists thousands of pages does not fit a context window. The agent greps, misses, and greps again. Each loop spends thinking tokens and another fetch. The map failed before a useful page was read.

Cloudflare documented the failure mode on their own docs and the change that followed. They generate a separate `llms.txt` per top-level directory and keep a root file that points at those directories. They omit roughly 450 directory-listing pages that only repeated child links. They enrich each entry with titles and descriptions already present in docs frontmatter. Then they pointed one agent, Kimi-k2.5 via OpenCode, at Cloudflare’s docs and at other large technical docs. On average, the agent on Cloudflare’s documentation used 31% fewer tokens and reached the correct answer 66% faster than on the average site that had not been refined the same way ([Introducing the Agent Readiness score](https://blog.cloudflare.com/agent-readiness/), 17 April 2026). That comparison is theirs, on that agent profile. It is a reason to inspect the map your agent actually opens. It is not a forecast that your next split file will post the same percentages.

Markdown negotiation is the companion measurement on the same post. In some cases, markdown instead of HTML cut tokens by up to 80% on their pages. Again, their measurement. The optimization lesson for a dead step lives in [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization). The documentation lesson here is simpler: if the index cannot be scanned, the evidence pages never get a fair chance.

## Presence of the file is not use of the file

Publishing `llms.txt` answers whether a public agent can reach the map. It does not answer whether any agent opened it last week, followed a link, or finished a job with it. [A machine-readable file can exist and still be ignored](https://www.agentrail.co/blog/llms-txt-for-ai-agents) owns that presence-versus-use gap, including the Ahrefs log study that shows how often valid files see no traffic. Use that essay when the question is “did anyone fetch it?” Stay here when the question is “what should they read when they do?”

The practical consequence is an audit order. Confirm the file is public and well formed. Then measure which tasks requested it and which pages followed. A green readiness check clears the first gate. It does not clear the second.

## Connect the opened URLs to a name or a finished task

A recommendation and a finished task leave different end states. They often share a reading trail.

On an unaided preference run, the prompt names a job and not your product. The agent may search, open an index, fetch a quick start, and then install a package. The documentation for AI agents in that run is every URL opened before the product name appears in code or prose. If your competitor’s markdown loaded and yours never did, the preference problem starts in the reading list, not in a pricing page. How to design that test is the [AI agent recommendation test](https://www.agentrail.co/blog/ai-agent-recommendation-test). Why analytics often miss the choice is [coding agent recommendations](https://www.agentrail.co/blog/coding-agent-recommendations).

On a task that already pointed at you, the trail continues past the name. Docs, then SDK install, then API or MCP call, then auth or outcome. The opened docs still matter. An outdated quick start that points at a deprecated route can “recommend” you and still fail the task. Connecting those steps into one record is journey work. Measuring the outcome is measurement work. The documentation job is to make sure the pages on that path carry current facts in a form the agent can consume.

Either way, write down the URLs. A vague report that “docs were confusing” is not actionable. A list that says the agent opened `/llms.txt`, then `/docs/email.md`, then never opened `/openapi.json` is actionable.

| Stage | What the agent opens | What you learn |
| --- | --- | --- |
| Index | `/llms.txt` or a scoped twin | Did the map get a chance? |
| Evidence | Markdown quick start, guide, or reference | Did the right claim load? |
| Contract | OpenAPI or equivalent | Could the call be formed correctly? |
| Outcome | Install, name, or verified state | Did the read become a pick or a finish? |

## Audit the path you want agents to take

Start with a job written in developer words, not marketing words. Run it on a stated agent profile. List every documentation URL opened before the agent names a product, installs a package, or stops.

Check the map. Prefer a root or scoped `llms.txt` whose entries carry semantic names and short notes. Prefer links that land on markdown, not on directory listings. Prefer current install and auth examples in the first page the index advertises for that job.

Check the evidence. If the fact lives only after JavaScript runs, many agents never see it. Put the fact in the HTML, the markdown twin, or the linked spec. If the job needs an API, link the OpenAPI document from the same index the agent already trusts.

Check use. Confirm the paths appear in the trajectory you care about. If the file never loads, you still have a presence or discoverability problem. If it loads and the agent still names someone else, the content or the competitive context is the next question, not another checkbox that the file exists.

A public [agent readiness](https://www.agentrail.co/agent-readiness) scan is a fast way to see which declared surfaces are reachable before you invest in a longer preference or task run. It does not replace the audit of what a real prompt opened.

## Frequently asked questions

### Is HTML documentation enough for agents?

Sometimes an agent can extract meaning from HTML. It still pays more tokens for chrome, and some facts never appear without JavaScript. Markdown twins and a short index reduce that cost. HTML alone is a weak default when you care about recommendation or task completion.

### Do we need both a root llms.txt and per-product files?

Large sites usually need both. A root file that only points at scoped indexes keeps the first context window usable. Per-product files hold the page lists. Cloudflare’s public docs pattern is the clearest large-site example. Small sites can start with one curated file if it stays short.

### How is this different from agent discoverability?

Discoverability asks whether an agent can find the docs, the API, and the MCP server. This page assumes the agent is already looking and asks what it should open next, and how those opens connect to a name or a finished task. Use [agent discoverability](https://www.agentrail.co/blog/agent-discoverability) for the doorway. Use this page for the reading list behind it.

### When do we measure preference versus task completion?

Measure preference when the prompt does not name you and you care which vendor gets chosen. Measure task completion when the agent is already on your surfaces and you care whether the job finished. Both measurements can inspect the same documentation trail. They answer different questions.

## Shape the files agents open, then prove they opened them

Documentation for AI agents is not a second docs product for humans. It is the index and the evidence pages that appear in the agent’s fetch list before a recommendation or a stop. Keep the map scannable. Keep the pages in markdown the agent can use. Link the contracts the job requires. Then connect those URLs to an unaided pick or a verified outcome.

When you want to test the pick without putting your brand in the first question, use the [recommendation test method](https://www.agentrail.co/blog/ai-agent-recommendation-test). When you want a public check that the surfaces are declared, start at [agent readiness](https://www.agentrail.co/agent-readiness). Pricing for journeys, simulations, and included readiness scans is on [pricing](https://www.agentrail.co/pricing).