--- title: "Agent discoverability" description: "Agent discoverability is whether an agent can find your docs, your API, and your MCP server without a person pasting the URL." canonical: "https://www.agentrail.co/blog/agent-discoverability" author: "Sidhant Goyal" published: "2026-10-17T00:00:00Z" last_updated: "2026-10-22T18:13:14.787Z" --- # Agent discoverability Agent discoverability is whether an agent can find your docs, your API, and your MCP server without a person pasting the URL. ![Search icon over a short list on a warm brown blur](https://images.ctfassets.net/1htfyfnn1o5y/1sOhM2mMBNLFqZN1gHVQ7n/a21e827332451d69742b60b0a603b19c/hero_image-3.png?w=1600&fm=webp&q=72) An agent can only use what it can find. If the docs sit behind a click-path, the task stops before it starts. A private API contract, or an MCP server that exists only in a screenshot, stops it the same way. Agent discoverability is that finding problem. It asks whether an agent can locate your documentation, your API, and your MCP server from the public web, without a person handing over the URL. > **Key Takeaways** > > Agent discoverability is whether an agent can find the docs, the API, and the MCP server on its own. > > Docs are discoverable when a short markdown index, such as `llms.txt`, points at markdown pages. A sitemap is a different file. > > An API is discoverable when a machine-readable contract is linked, including from `/.well-known/api-catalog` as RFC 9727 defines it. > > An MCP server is discoverable in two steps: a static card before connect, and `tools/list` after connect. The card deliberately omits the tool list. > > The check is a public scan of what an agent can reach. The scan lives at [agent readiness](https://www.agentrail.co/agent-readiness). ## Agent discoverability asks whether an agent can find the thing it was sent for Agent discoverability is the degree to which an agent can locate a product’s public docs, API description, and MCP server without prior knowledge of their URLs. Three tests follow from that definition. Can it find the documentation that answers the task? Can it find a machine-readable description of the API? Can it find where to connect for MCP, and then which tools exist for the caller it is using? A human finding the same pages through the nav is not the test. Humans tolerate menus. Agents fetch. A page that exists only after a client-side render, or only after login, is absent for a fetch that does not run that path. Discoverability is also not citation. A model can mention your product from memory and still fail to open the current docs. [Agent experience](https://www.agentrail.co/blog/agent-experience) covers what happens after the agent is in the product. This page stops at the doorway. Being indexed by a search engine is related and not sufficient. Search ranking sends some agents to you. It does not tell them which URL is the API contract, or which host speaks MCP. ## Docs are discoverable when a short index points at markdown The file built for this job is [`llms.txt`](https://llmstxt.org/), proposed by Jeremy Howard. The page was first published on 3 September 2024 and updated on 10 August 2026. The format is markdown at `/llms.txt`, or at a deeper path such as `/docs/llms.txt`. A file covers the URLs under its path. The most specific file wins. The only required section is an H1 with the name of the site or project. A blockquote can summarize it. H2 sections then hold lists of links. Each item is a markdown link, optionally followed by a colon and a note. An “Optional” section is the convention for links an agent may skip when context is tight. Howard’s instruction to agents is direct. View or search `llms.txt`, then follow the relevant links. Those links should point at LLM-friendly content, not at another HTML shell. He also proposes a markdown twin of pages that matter, at the same URL with `.md` appended or with the extension replaced. A link relation can advertise it: `rel="alternate"` with type `text/markdown`. A sitemap does not do this job. Howard lists why. Sitemaps often omit the markdown URLs. They do not point at useful external references. In aggregate they are too large for a context window, and they include pages the agent does not need. `llms.txt` is a curated overview. The sitemap is an index for search crawlers. Coding agents already follow this pattern on documentation sites. The proposal says the heaviest use is software docs, where an agent needs an API reference or a tutorial and does not want the chrome around it. ## An API is discoverable when a machine-readable contract is linked Docs can describe an API in prose. An agent can still mis-read prose. A contract is better. [The OpenAPI Specification](https://spec.openapis.org/oas/latest.html) defines a language-agnostic interface description for HTTP APIs. Its own wording: humans and computers should be able to discover and understand the service without source code, without additional documentation, and without inspecting network traffic. When the description is actually correct, a client can call the service with a small amount of logic. Spec 3.2.1 is dated 10 September 2026. Discoverable means the file is at a stable URL and something public points at it. Common placements are `/openapi.json` or a link from `llms.txt`. The exact path matters less than the link. An undocumented filename that only your deploy script knows is not discovery. There is also an Internet standard for the pointer. [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727) defines a well-known URI, `api-catalog`, and a link relation of the same name. A request to that resource returns a document of links and metadata for the publisher’s APIs. The RFC’s goal is automated discovery of public API endpoints. The catalog itself can link to other catalogs. Publishers can also emit `rel=api-catalog` from an ordinary response. The well-known path is `/.well-known/api-catalog`. That is the place an agent can try when it knows the origin and does not know your directory layout. The RFC body does not state a publication month, so this article does not invent one. The mechanism is what matters. Also say, in plain words, where a key is created and which header carries it. An agent will copy the example you publish. ## An MCP server is discoverable in two steps MCP discovery is easy to oversimplify. There is a static step and a live step, and they are not interchangeable. [SEP-2127, MCP Server Cards](https://modelcontextprotocol.io/seps/2127-mcp-server-cards), was created on 21 January 2026 and is marked Final. It adds an optional document that describes a remote server before a client connects: identity, transport URLs, and supported protocol versions. Cards may live at any unreserved URI. The reserved recommendation is a path ending in `/server-card`. For a whole domain, an AI Catalog at `/.well-known/ai-catalog.json` can link or embed those cards. The SEP points at a well-known URI pattern for the catalog. The card is public metadata. The SEP says it must not carry credentials, internal topology, or session data. Serve it over HTTPS. The card does not list tools. That omission is intentional. Tools, resources, and prompts can change with the caller’s identity, feature flags, and deployment. A static file would go stale, and a client that trusted it could be wrong. The SEP says there is no substitute for the live list operations: `tools/list`, `resources/list`, and `prompts/list`, made with the caller’s credentials. Capabilities and extensions stay out of the card for the same reason. If a card and a live `server/discover` result disagree, the SEP says clients should prefer the live values. So the discoverability test for MCP is both questions. Can an agent find where to connect, from a card or a catalog, without a human pasting a URL? After it connects, can it list the tools this caller is actually allowed to use? Recording what happens on the call is a different job. [MCP observability](https://www.agentrail.co/blog/mcp-observability) covers what to store when a tool runs. Finding the server is the step before that log exists. | Surface | What “found” means | Where an agent can look | | --- | --- | --- | | Docs | A short index points at markdown pages | `/llms.txt` or `/docs/llms.txt` | | API | A machine-readable contract is linked | OpenAPI URL, or `/.well-known/api-catalog` | | MCP | A connect target is public, then tools are listed live | A server card or AI Catalog, then `tools/list` | ## A repo has its own front door Some agents never start on your marketing site. They start in a repository the buyer already cloned. `AGENTS.md` is the convention for that case. [The project site](https://agents.md/), retrieved 17 October 2026, says it is used by over 60,000 open-source projects. It is standard markdown, with no required fields. The closest file to the code being edited wins when several exist. Put in the commands you would give a new teammate: install, test, and any directory to avoid. If you list test commands, the agent will try to run them. `AGENTS.md` does not publish your hosted API, and it does not replace a server card. It answers a coding agent that is already inside the repo. The product-site problem is still `llms.txt`, the API catalog, and the MCP card. Keep the two front doors, and do not make one of them a copy of your homepage. ## Published and blocked is still hidden A file that exists and cannot be fetched is not discoverable. `robots.txt` and `llms.txt` are not substitutes. Howard’s proposal says `robots.txt` tells automated tools what access is acceptable, including for indexing bots. `llms.txt` is read on demand, when an agent needs a fact while helping a user. You can allow one and refuse the other. If you publish `llms.txt` and your robots policy blocks the fetch the agent actually makes, the index does not help. Decide which outcome you mean. JavaScript is the other quiet block. Howard’s background is blunt about it. Pages are built for people. Turning them back into clean text is difficult and imprecise. An agent that only receives the empty shell of a single-page app has not found your docs, even if a browser with a long wait would paint them. Markdown at the same URL, or server-rendered text, is the fix. The index file points at that text. It does not render the application for you. Login is a third block, and it is legitimate. Private APIs should stay private. Discoverability is about what you intend agents to use without an account, plus a clear public explanation of how an account is created when the task needs one. Hiding the signup docs behind the signup defeats the task. What the agent does after it gets in is [agent behavior](https://www.agentrail.co/blog/agent-behavior). Repeated fetches of a blocked URL are a path. The missing file is the cause. ## A public scan is the check You can hold the three questions without building a detector. Agent Readiness, on [the public leaderboard](https://www.agentrail.co/agent-readiness), scans a site the way an agent can reach it. The page says the scan does not log in and does not call a private API. If an agent on the public web cannot see it, it is not in the score. The checks cover what an agent can find, read, use, and pay for. The page names Discovery, Access and Usability, plus payments checks. That is the right next step after this article. The definition of readiness, as a concept, is in [what agent readiness is](https://www.agentrail.co/blog/what-is-agent-readiness). This page does not restate it, and it does not walk the checker. Scores on the board move every week. Quoting a rank here would be stale on purpose. Run the scan against the three surfaces above. If the docs index is missing, the API catalog 404s, or nothing tells an agent where MCP connects, the scan is how you see that from the outside. We include readiness with the product. The reading of real tasks, after something is found, belongs to [agent experience](https://www.agentrail.co/blog/agent-experience). ## FAQs ### What is agent discoverability? Agent discoverability is whether an agent can find your docs, your API description, and your MCP server without a person supplying the URLs. It is a property of those public surfaces, not a rank in a chat answer. ### Does llms.txt replace my sitemap? No. A sitemap lists pages for search indexing. `llms.txt` is a short, curated map for an agent that needs a few markdown URLs. Howard’s spec treats them as different files with different jobs. ### Should an MCP server card list every tool? No. SEP-2127 leaves tools, resources, and prompts off the card because they change with identity and deployment. The card helps a client find where to connect. `tools/list`, with the caller’s credentials, is the live list. ### Is a JavaScript docs site discoverable? Only to an agent that renders it. A fetch that receives an empty application shell has not found the docs. Publish markdown, or server-render the text, and point `llms.txt` at those URLs. ## Conclusion An agent can find your product when three public answers exist: a docs index, an API contract, and an MCP connect target plus a live tool list. Menus and memory are not those answers. Check the doorway with the same constraint the agent has. No login, no private API, no assumed URL. [Agent readiness](https://www.agentrail.co/agent-readiness) runs that scan. Start there, then fix the file the task could not open.