---
title: "Agent Analytics Platforms: What to Look For"
description: "What to look for in an agent analytics platform: production tasks across docs, APIs, SDKs, and MCPs. A buying test from Agentrail, with no vendor ranking."
canonical: "https://www.agentrail.co/blog/agent-analytics-platforms"
author: "Vivek Mittal"
published: "2026-10-17T00:00:00Z"
last_updated: "2026-10-23T10:23:08.303Z"
---

# Agent Analytics Platforms: What to Look For

What to look for in an agent analytics platform: production tasks across docs, APIs, SDKs, and MCPs. A buying test from Agentrail, with no vendor ranking.

![Eye icon on a dark blur with a warm glow](https://images.ctfassets.net/1htfyfnn1o5y/pdf3TqVp5fMrT7xVIKk3b/18730c454603a4c66a2bdd3bce7de6df/hero_image-1.png?w=1600&fm=webp&q=72)

Agentrail publishes this guide. We sell software in the category, so the criteria below are the ones you should apply to us as well as to anyone else. An agent analytics platform shows what an AI agent tried to do on your product, the path it took, and where that path stopped.

The pages that rank for nearby queries usually describe a tracer for an agent you operate. You install an SDK, every model call and tool call becomes a span, and you debug that process. That is a real product, and many teams need it. It answers a different question from the one in this guide. Here the buyer wants agents that show up on their own: on the docs, the API, the SDK, and the MCP server.

Those agents are already in production at a scale you can cite. In December 2025, [Anthropic reported](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation) more than 10,000 active public MCP servers, and more than 97 million monthly SDK downloads across Python and TypeScript. The same week, [GitHub reported](https://github.blog/open-source/maintainers/mcp-joins-the-linux-foundation-what-this-means-for-developers-building-the-next-era-of-ai-tools-and-agents/) that the MCP repository had reached 37,000 stars in under eight months. GitHub also reported more than one million pull requests from Copilot's coding agent in its first five months. Some of those agents are yours. Many of them call someone else's docs and tools.

[Agent analytics](https://www.agentrail.co/blog/agent-analytics) is the category name for that second view. This page is the buying test.

> **Key Takeaways**
>
> Name the population first. A tracer for agents you run is a different purchase from a view of agents that arrive on your docs, API, SDK, and MCP.
>
> The passing demo is one task with steps on more than one of those surfaces, taken from production traffic.
>
> A finished tool call is not a finished task. Ask where the task stopped, and for the evidence at that step.
>
> Client names are labels. Retention, export, and the unit of price have to be written down before you compare vendors.
>
> Agentrail is the publisher. Run the same proof on our product. The checklist has no rank.

## Decide which agent you can actually see

There are two populations, and a platform that covers one can still be the wrong buy for the other.

The first population is agents you build and host. You can add a tracer and read every model call. [LangSmith's observability docs](https://docs.langchain.com/langsmith/observability-concepts) describe that record. A run is one unit of work, such as a model call or a tool call. A trace is the tree of runs for one operation. A thread groups traces when you pass `thread_id`. A trajectory flattens the thread into messages. Data comes from a framework integration, or from manual instrumentation. That shape fits when the agent process is yours.

The second population never gives you that process. It fetches a doc, tries an SDK snippet, calls an HTTP API, and may call your MCP server. You see those requests at your edge. You don't see the model weights or the prompt inside someone else's product. A buying conversation that starts with "which framework do you trace?" has already assumed the first population.

Ask the vendor which population the demo uses. If every screen is a trace from an app they deployed, you are looking at the first product. Keep it on the shortlist when you operate agents. Keep looking when the user is an outside agent on your surfaces.

LangSmith also documents two limits worth asking every tracing vendor, not only them. SaaS retains trace data for 180 days from ingestion, and then deletes it. A single trace holds at most 25,000 runs. Those numbers are from the [observability concepts page](https://docs.langchain.com/langsmith/observability-concepts). Another vendor may keep data longer or cap traces differently. The question is the same. How long do production tasks stay queryable, and what happens when a long task exceeds the cap?

## Ask for one task across docs, APIs, SDKs, and MCPs

A platform passes this criterion when it can show one task that touched more than one surface. The surfaces that matter here are the website, the docs, the API, the SDK, and MCP. A gallery of MCP tool names, with no doc view and no API call, fails the test even when the tool chart is excellent.

MCP-only capture is a useful half. PostHog's MCP analytics records a `$mcp_tool_call` after you install the SDK in your server. The [event reference](https://posthog.com/docs/mcp-analytics/events) and the [sample queries](https://posthog.com/docs/mcp-analytics/queries) cover tool name, error flag, duration, client label, and tools listed but never called. That stream starts at the MCP call. The doc page read before the connection is a separate capture. So is the REST call, unless another pipeline stored it. You still have to see the join.

The join is the product. A task that dies on a docs page never produces a tool call, so an MCP chart stays quiet while the task fails. A docs success followed by a failed `tools/call` needs both steps on screen before you pick a fix. Ask the vendor to walk one task in order: doc or page, API or SDK call, MCP tool, and the step that stopped.

What to record on the MCP step is its own spec. Use [MCP observability](https://www.agentrail.co/blog/mcp-observability) as the field list when you audit that part of the demo. Which of those tools get called, and which were listed and never called, is [MCP analytics](https://www.agentrail.co/blog/mcp-analytics). A platform that can't answer either question from production rows is showing a diagram. Ask for the rows.

## Separate a finished call from a finished task

Span status and tool status are real signals. An HTTP 200 means the handler returned. `isError: false` means the tool did not flag an execution error. The agent can still hold the wrong identifier, an empty list, or a doc it could not apply.

Ask where the task stopped, and for the evidence on that step. Evidence is a URL, a status code, a tool name, an outcome class, argument names, or a result size. A line that says the agent was confused is a hypothesis. Keep it off the purchase decision.

The MCP outcome classes are worth requiring by name. A protocol error, a tool execution error, a completed result, a pause for more input, and a local timeout are different endings. A platform that collapses them into "success" and "error" will send you to the wrong fix. The field rules are in the observability article. The way to count retries and listed-but-unused tools is in the analytics article. This buying guide only checks that the platform can show them on a task, next to the docs and API steps.

Task completion, as a rate, has its own definition elsewhere in this cluster. In a proof, you do not need the formula. You need the vendor to point at the last step they believe finished the job, and to show you the row.

## Treat the client name as a label

On current MCP, `io.modelcontextprotocol/clientInfo` is optional and self-reported. The [base protocol](https://modelcontextprotocol.io/specification/2026-07-28/basic) says the value is for display, logging, and debugging. Implementations should not use it for security decisions. A platform that blocks or authorizes from that string is outside the spec.

The string is also a weak chart key. PostHog's event reference shows a User-Agent splitting clients that share a name, including several Claude Code builds. Claude.ai on the web, desktop, and mobile can collapse to one label. Stdio has no User-Agent. A bucket called "Other" means the label was missing. Ask to see that bucket.

Grouping many requests into one task takes more than the client string. We combine request, behavioral, temporal, and product-level signals to decide which interactions belong together. The method and its limits are in [agent fingerprinting](https://www.agentrail.co/blog/agent-fingerprinting). Ask any vendor who quotes an accuracy figure for the definition and the sample.

Revision `2026-07-28` also removed protocol sessions. A platform that stitches MCP calls by `Mcp-Session-Id` alone will drop current clients, or it will invent a session from the TCP connection. The base protocol says a connection is not a conversation. The proof should include a modern, stateless call, correlated by a trace id or by a handle the tool actually returned.

## Compare retention, export, and the unit of price

Compare prices only after you know the unit and the retention.

Ask four questions, and write the answers down:

1. What is the unit? A trace, a seat, an event, a session, or a journey?
2. How long do production tasks stay queryable, and what is deleted after that?
3. Can you export the rows, or only see them in the vendor UI?
4. What happens at the cap? LangSmith's 25,000-run trace limit is one example. Ask where the extra steps go.

A seat price and a per-journey price match only after you know how many journeys a seat stood in for. A short retention window hides the quarter you will want back.

Evals are a separate line. A scripted agent in a sandbox tells you whether a change holds on a task you wrote. Production tasks tell you what unscripted agents already did. Most teams want both. A vendor that only demos the sandbox is selling the eval. Buy it if you need it. Call it an eval in the notes so the renewal conversation stays honest.

| Criterion | Ask for this | A weak answer |
| --- | --- | --- |
| Population | Tasks from agents you did not instrument | Only traces from an app the vendor hosts |
| Surfaces | One task across docs, API or SDK, and MCP | A single-surface chart |
| Task ending | The step that stopped, with a row of evidence | "The agent was confused," and no row |
| MCP outcome | Protocol error, tool error, complete, input required, timeout | One success bit |
| Client label | The raw label, plus an "unknown" bucket | Three tidy names and no missing data |
| Correlation | A modern call joined without a protocol session | `Mcp-Session-Id` as the only key |
| Retention and export | A stated window and a way to leave with the rows | "Contact sales" as the only retention answer |
| Price unit | Journey, trace, event, or seat, defined | A number with no unit |
| Eval versus production | Both, labeled | A sandbox run described as live traffic |

The table is a checklist. It is not a score, and it has no vendor column.

## Run the proof on a task you recognize

Pick a task a real agent would attempt against your product. "Create a resource," "look up an account," or "ship the snippet in the quickstart" is enough. You want a task your own engineers can judge.

Then require the vendor to show it under these rules:

1. The task comes from production traffic, or the vendor says the only copy is a simulation.
2. At least two surfaces appear, in order. A doc step plus an API, SDK, or MCP step is the minimum.
3. The MCP step matches the field list in [MCP observability](https://www.agentrail.co/blog/mcp-observability): tool name, outcome, protocol version, and a client label or an explicit "absent."
4. The vendor can say which tools were listed and never called, using the cuts in [MCP analytics](https://www.agentrail.co/blog/mcp-analytics). Missing list records get said out loud.
5. Retention, export, and the price unit match the order form.
6. A scripted run is labeled as an eval. It may sit beside the production task. It does not replace it.

Fail the proof when the demo cannot leave the vendor's sample application, or when every agent in the deck installed their SDK. Outside agents will not install it. Note which population you actually saw.

## What the publisher sells, under the same test

We are Agentrail. The legal entity is Aeternus Inc. We reconstruct how AI agents discover, choose, and use a product across website, docs, APIs, SDKs, and MCPs. You should see the task, the path, the break, and a fix. Company background is on the [about page](https://www.agentrail.co/about).

Public prices are on the [pricing page](https://www.agentrail.co/pricing):

- Agent journeys: 100 free per organization each month, then $0.10 per journey. A journey is the production reconstruction across those surfaces.
- Simulations, also called evals: 10 free runs each month, then $1 per run. Coding agents run in sandboxes against preference and experience prompts.
- Agent Readiness is included. It scans the website, docs, and app the way agents see them, with and without JavaScript, and returns findings with evidence and a fix prompt.
- Recommendations are included. They rank fixes from journeys, scans, and simulations.
- There is no seat fee. A spend cap is available. You can cancel anytime, and the free tier stays.

Simulations and journeys are different products. One is a task you script. The other is traffic you already have. The proof asks us to show both, labeled. A simulation alone is the eval.

## Adjacent categories answer a different question

A few adjacent categories use overlapping words. They answer a different question.

Citation and visibility tools, including Profound, Gauge, and Ahrefs' visibility products, report whether models mention or cite you. That is a distribution question. This guide is about whether an agent completed a task on your docs, API, SDK, or MCP.

Contact-center products from Genesys and Salesforce use "agent experience" for human agents on a service desk. The AI-agent meaning is [agent experience](https://www.agentrail.co/blog/agent-experience).

An "analytics agent" is a copilot that queries your warehouse. It is a different search. The category this page buys against is defined in [agent analytics](https://www.agentrail.co/blog/agent-analytics).

## FAQ

### What is an agent analytics platform?

An agent analytics platform shows the tasks AI agents attempt on your product and where those tasks stop. For this guide, the product surfaces are the website, the docs, the API, the SDK, and MCP. A tracer for an agent you host can be part of the stack. It is not the whole test.

### How is this different from LLM observability?

LLM observability records runs inside an agent you instrument: model calls, tool calls, and latency. LangSmith is one example, with traces, threads, 180-day SaaS retention, and a 25,000-run cap per trace. Agent analytics, as this page uses the term, includes agents that never ran in your process, across more than one of your surfaces.

### What should the proof include?

One task you recognize, from production if the vendor has it, with at least two surfaces in order. The MCP step should expose tool name, outcome, protocol version, and client label. The vendor should be able to say which listed tools were never called. Retention, export, and the price unit should match the order form. A simulation should be labeled as a simulation.

### Does Agentrail rank the platforms on this page?

This page has no ranking. Agentrail publishes the guide and sells a product you can put through the same proof. The checklist has no vendor column. Prices we quote for ourselves are the public ones on the pricing page.

## Buy the view you can check

Choose the population, then make the vendor show one task across the surfaces that population actually touches. Require the MCP fields, the unused-tool cut, a stated retention window, and a price unit you can forecast. Put Agentrail through that proof the same way. The category definition sits on [agent analytics](https://www.agentrail.co/blog/agent-analytics), and the meter sits on [pricing](https://www.agentrail.co/pricing).