---
title: "A Machine-Readable File Can Exist and Still Be Ignored"
description: "Publishing llms.txt, OpenAPI, or an MCP card is a presence check. Ahrefs found most valid files got zero traffic in May 2026. Use is a later measurement."
canonical: "https://www.agentrail.co/blog/llms-txt-for-ai-agents"
author: "Sidhant Goyal"
published: "2026-11-02T00:00:00Z"
last_updated: "2026-11-02T06:30:31.063Z"
---

# A Machine-Readable File Can Exist and Still Be Ignored

Publishing llms.txt, OpenAPI, or an MCP card is a presence check. Ahrefs found most valid files got zero traffic in May 2026. Use is a later measurement.

![White clipboard with a checkmark on a green textured gradient](https://images.ctfassets.net/1htfyfnn1o5y/Gx9BvT6cVr5pWHmHj2Uau/c89eb8551c219486a5f2fba96a6f8d9c/bdcef39f-0906-4cf8-aa0d-448d299f7d45.png?w=1600&fm=webp&q=72)

A machine-readable file can return HTTP 200 and still change nothing about how an agent works. Publishing `llms.txt`, an OpenAPI document, or an MCP card answers one question: is the surface on the public web? It does not answer whether any agent fetched that surface, followed its links, or finished a task with it.

That split is the job of this essay. [Agent discoverability](https://www.agentrail.co/blog/agent-discoverability) covers how an agent finds docs, an API, and an MCP server. [Agent readiness](https://www.agentrail.co/blog/what-is-agent-readiness) defines the public contract a scan can check. Here the claim is narrower. Presence is the first check. Use is later. The public evidence for `llms.txt` comes from [Ahrefs’ May 2026 log study](https://ahrefs.com/blog/llmstxt-study/). The same pattern shows up when an MCP server is live and still never appears in organic sessions.

> **Key Takeaways**
>
> • A reachable `llms.txt`, OpenAPI file, or MCP card is a presence check. It is not proof an agent used it.
>
> • In Ahrefs’ study of 137K domains, 28% published `llms.txt`, and 97% of valid files got zero requests in May 2026.
>
> • AI bots did not hunt missing files. Agents fetch `llms.txt` when a link or instruction points at it.
>
> • An AX audit of Freestyle found a docs MCP that helped once installed, and that never appeared in thirteen organic discoverability sessions.
>
> • Confirm the file is public, then measure whether tasks fetch it. Start with the [agent readiness](https://www.agentrail.co/agent-readiness) scan.

## Presence is the first check, not the last

Presence asks whether a public agent can reach the contract you meant to ship. A readiness scan is built for that question. It looks for declared surfaces on the website, docs, and app, with and without JavaScript, and without a login.

A green check on that scan means the file or interface was there when the crawler asked. It does not mean a coding agent opened it last Tuesday. It does not mean the agent trusted the links inside. It does not mean the task finished.

Teams often stop at the green check. The file is cheap to generate. Docs platforms emit one by default. A Lighthouse audit can nag for it. None of those events are a use event. [What agent readiness is](https://www.agentrail.co/blog/what-is-agent-readiness) keeps the scan as a precondition on purpose. This page stays on the next gap: the file can exist and still be ignored.

## Most published llms.txt files get no traffic

[Ahrefs analyzed](https://ahrefs.com/blog/llmstxt-study/) every domain in Ahrefs Web Analytics that received traffic in May 2026: 137,210 domains. They checked each root for an `llms.txt` that returned HTTP 200 and was real markdown, then classified every request to `/llms.txt` paths.

Two headline numbers matter for this essay. First, 28% of those domains published a valid file. Ahrefs notes the sample skews technical and SEO-aware, so treat 28% as an upper bound for the wider web. Second, of the roughly 38,000 domains with a valid file, 97% received zero requests for it in May 2026. No bots. No humans. Nothing.

The remaining 3%, about 1.1K domains, received all of the measured `llms.txt` traffic. If you publish the file today, the base rate says nothing will fetch it. That is a presence success paired with a use failure at population scale.

Ahrefs is also clear about a ceiling. A request means the file was fetched. It does not mean the bot acted on the contents. Every share they report is therefore an upper bound on real consumption. Write “fetched,” not “followed every link,” unless you have a second measurement.

## AI bots do not hunt a file that is missing

The same study checked requests that returned 404 for `/llms.txt`. Valid files drew mostly bot traffic. Missing files drew mostly human traffic. The AI-bot share of those 404s was zero.

That split kills a common hope. Publishing `llms.txt` does not put you on an AI radar that was already probing for the path. Ahrefs’ conclusion is direct: AI tools fetch `llms.txt` when a link, an index, or a user instruction tells them it exists. They do not go looking for files that are not there.

So a silent log is ambiguous. Either nothing knows to ask, or nothing needs the file for the tasks that hit your site. Presence alone cannot tell those two stories apart. You still need a use measurement: which tasks requested the path, and what they did next.

## What llms.txt is for when an agent does open it

[`llms.txt`](https://llmstxt.org/) is a short markdown index. Jeremy Howard’s proposal, first published 3 September 2024 and updated 10 August 2026, places it at `/llms.txt` or at a path such as `/docs/llms.txt`. The file covers URLs under that path. An H1 with the project name is required. H2 sections list curated links to LLM-friendly content, usually markdown twins of important pages.

The format is not a sitemap and not a robots directive. Howard’s page says agents should view or search `llms.txt`, then follow the relevant links. The file stays small enough for a context window. Detail lives behind the links.

When something does fetch the file, Ahrefs’ bot mix is closer to that design than to AI-search marketing. Across the 3% of files that got any traffic, 96% of requests came from bots. Named AI tools accounted for 19.5% of requests. AI agents and agentic infrastructure were 10.5%, ahead of AI retrieval bots at 1.1%. Claude-Code out-fetched every AI retrieval bot, every AI assistant, and every AI training crawler in their ranking of named agents. GPTBot led training crawlers.

That pattern matches the intended job: a coding or task agent that was pointed at your docs, not a retrieval bot building an Overview. It still does not rescue the 97%. Most files never enter that funnel.

How an agent finds the index in the first place is [agent discoverability](https://www.agentrail.co/blog/agent-discoverability). What it actually opens before it names you is [documentation for AI agents](https://www.agentrail.co/blog/documentation-for-ai-agents). This page does not retell either how-to.

## OpenAPI and MCP cards fail the same way

`llms.txt` is the file with a large public fetch study. OpenAPI and MCP metadata fail through the same logic even when the population study is smaller.

An OpenAPI document that sits at a stable URL and is linked from docs is present. An agent that already has a wrong endpoint in context may never request it. An MCP server card or hosted docs MCP can be correct, documented, and still absent from the sessions that matter.

[Tech Stackups’ Freestyle AX audit](https://techstackups.com/articles/freestyle-ax-audit/), published 8 April 2026, is a concrete case. Freestyle shipped a hosted docs MCP and an `llms.txt`. Once the MCP was installed, the agent used it and completed integration tasks with less diagnostic thrash. In the thirteen organic discoverability sessions, that MCP never appeared. The audit’s agent-tooling summary states the tooling was present and effective, and that the MCP was not organically discoverable.

The product review is not the point. The measurement is. Presence of the MCP was true. Organic use in discovery sessions was false. Installing the server closed the gap for a scripted task. It did not close the gap for unaided finding. Teams that only ask “do we have an MCP?” stop one step early.

## Why agents skip a file that exists

Agents work from goals, tools, and the URLs those tools already hold. They do not crawl your origin the way a search indexer does. Ahrefs’ 404 data is the population version of that behavior: no AI bot showed up asking for a missing `llms.txt`.

Several product failures follow.

The file is unlinked. Nothing in HTML, docs, or agent instructions points at it, so nothing fetches it.

The fetch is allowed and the useful pages are not. `llms.txt` points at markdown that is blocked, client-rendered empty, or behind login. The index was present. The payload was not usable.

The agent already has a competing source. A stale blog post, a cached answer, or a third-party comparison ranks higher in its search results than your root file.

The wrong path is published. Marketing ships `/llms.txt` while the coding agent works only from `/docs/` and never climbs to the root.

None of those failures show up as a missing checkbox on a naive “file exists” audit. They show up as zero requests, or as requests that never continue to the linked pages. [Agent discoverability](https://www.agentrail.co/blog/agent-discoverability) covers published-and-blocked cases. The use question still needs traffic and task data after the doorway is open.

## Measure use after you confirm presence

Treat the two checks as a sequence.

First, confirm presence with a public scan. The [agent readiness leaderboard](https://www.agentrail.co/agent-readiness) runs the same outside-in checks on declared surfaces. If `llms.txt`, OpenAPI, or MCP metadata is missing or unreachable, fix that before you debate models.

Second, ask whether any agent task fetched the file and followed what it pointed to. Server logs filtered to the path are a start. They still stop at fetch. Reconstructing the task across docs, API, SDK, and MCP is how you see whether the file changed the path or whether the agent abandoned the job one step later.

| Check | Question | Typical evidence |
| --- | --- | --- |
| Presence | Can a public agent reach the file or card? | HTTP 200, linked URL, readiness scan |
| Fetch | Did anything request it? | Access logs, bot analytics |
| Use | Did a task act on it and continue? | Linked resources fetched, task progress |

We include readiness scans with Agentrail so the presence check is cheap to repeat. Journeys reconstruct what an agent attempted across website, docs, APIs, SDKs, and MCPs when you need the use record. Public pricing for journeys is on [pricing](https://www.agentrail.co/pricing): 100 free journeys per organization each month, then $0.10 per journey.

## Frequently asked questions

### Does publishing llms.txt make AI search cite us?

Not on the evidence Ahrefs published. AI retrieval bots were 1.1% of requests to files that got any traffic at all, and 97% of valid files got no traffic in May 2026. Treat `llms.txt` as an on-demand index for agents that are already working with your site, not as a citation lever.

### How many sites publish llms.txt?

In Ahrefs Web Analytics for May 2026, 28% of 137K domains published a valid root file. The publisher calls that an upper bound relative to the whole web, because the sample leans technical.

### If the file gets zero traffic, should we delete it?

Not automatically. A quiet file is cheap. Deleting it removes the chance that a coding agent, pointed at your docs, will use it. Better moves: link it from docs and agent instructions, point it at real markdown, and watch whether tasks ever fetch it. Ahrefs also flags prompt-injection risk if the file becomes instruction-shaped. Keep it to plain links and descriptions you control.

### Is this the same as agent discoverability?

No. [Discoverability](https://www.agentrail.co/blog/agent-discoverability) asks whether an agent can find the docs, API, and MCP server. This essay asks whether a surface that is already findable and published is actually used. You can pass the first and fail the second.

## Conclusion

Shipping `llms.txt`, OpenAPI, or an MCP card clears the presence check. Ahrefs shows most valid `llms.txt` files still attract no requests. AI bots do not hunt the ones that are missing. An MCP can be installed and useful in a lab session and still never appear when the agent is not told to look.

Run the public check on [agent readiness](https://www.agentrail.co/agent-readiness). Read the definition in [what agent readiness is](https://www.agentrail.co/blog/what-is-agent-readiness) if you need the four questions behind the scan. Then measure use: which tasks opened the file, and whether [documentation for AI agents](https://www.agentrail.co/blog/documentation-for-ai-agents) was part of the path before a recommendation. Presence is necessary. It is not the finish line.