---
title: "How to Measure Agent Experience"
description: "Agent experience metrics cover task completion, the dead step, propensity, efficacy, repeatability, and path cost. They do not collapse into one score."
canonical: "https://www.agentrail.co/blog/how-to-measure-agent-experience"
author: "Harsh Raj"
published: "2026-10-17T00:00:00Z"
last_updated: "2026-10-22T09:23:44.417Z"
---

# How to Measure Agent Experience

Agent experience metrics cover task completion, the dead step, propensity, efficacy, repeatability, and path cost. They do not collapse into one score.

![Person with a vintage computer monitor for a head, typing among green leaves](https://images.ctfassets.net/1htfyfnn1o5y/10dt32c9LGJpcmBfDV96Yq/5cae445b1882b80940d062695a6d3396/hero_image-1.png?w=1600&fm=webp&q=72)

Agent experience analytics is how you stop arguing about a single percentage. Two teams can both say “success rate” and mean different events, different denominators, and different runs. The fix is a named set of measurements, each tied to a question you can act on.

[Agent experience](https://www.agentrail.co/blog/agent-experience) is the experience an AI agent has while discovering, choosing, and using your product. [Agent analytics](https://www.agentrail.co/blog/agent-analytics) is the practice of reconstructing that experience from real traffic. This page owns the measurements. It doesn't own the essay on what readiness means, and it doesn't own the edits you make after a number moves.

> **Key Takeaways**
>
> – Agent experience metrics are a set. They do not roll up into one agent-experience score.
>
> – Task completion checks the resulting state of the world, not the agent’s closing sentence.
>
> – The dead step, the retries, and the extra work are separate from the pass rate.
>
> – Propensity and efficacy answer two questions, and a single trial answers neither with much confidence.
>
> – Some of these numbers come from production journeys. Others require a task you script and repeat.

## Agent experience analytics measures the task

The unit is one agent task. An agent might read the docs, inspect an SDK, call an MCP tool, hit an API, and stop at auth, and your logs will show five events. The agent experienced one job. [Agent journey analytics](https://www.agentrail.co/blog/agent-journey-analytics) is the write-up of that stitch. We group the underlying requests with request, behavioural, temporal, and product-level signals. [Agent fingerprinting](https://www.agentrail.co/blog/agent-fingerprinting) covers those signals. We don't publish an accuracy rate for the grouping.

That unit is the thing [agent experience analytics](https://www.agentrail.co/blog/agent-analytics) reports. It is the agent’s path through your product. It is not a copilot that queries your warehouse, which is a different search entirely. It is also not the contact-center score for human support staff. If your dashboard was built for that staff, you are counting a different population.

## Agent experience metrics are a set

Agent experience metrics are the readings you keep side by side. Each one answers one question. None of them is a license to average the others.

- **Task completion.** Did the task reach a verified end state?
- **Terminal state.** Was that end a success, a partial, a refusal, or a failure?
- **Dead step.** Which surface, and which action, stopped the task?
- **Retries and extra steps.** How much work did the path take, including paths that finished?
- **Propensity.** When the task does not name you, does the agent find and choose you?
- **Efficacy.** When the task does name you, does the agent use the current interface correctly?
- **Repeatability.** Does the result hold across repeats, or only on a lucky run?
- **Path cost.** What did a completed task consume in steps, time, money, or human help?

A readiness profile sits beside this list. It asks whether the public surfaces are declared. [What is agent readiness?](https://www.agentrail.co/blog/what-is-agent-readiness) defines that check. A measurement essay updated 8 October 2026 is blunt about the limit. A declared-surface score does not predict success ([How to measure Agent Experience](https://agentexperience.tech/insights/measure-agent-experience/)). Keep it out of any weighted total. We don't publish a formula that folds these rows into one number, and you shouldn't invent one for a slide.

## Task completion checks the world

Task completion is whether the environment reached the goal state. Anthropic’s definition, in the 9 January 2026 evals essay, draws the line with a booking. The agent can say the flight is booked. The outcome is whether a reservation exists in the database ([Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)). NVIDIA states the same gate as a ratio: successful tasks divided by tasks. Call accuracy can look fine while a later write never lands ([How to Evaluate AI Agents](https://developer.nvidia.com/blog/how-to-evaluate-ai-agents-from-tool-calls-to-task-completion/), 21 September 2026).

Grade the end state when you can check it. A required sequence of tool names is a brittle grader. Agents regularly take a valid path the test author did not list. Anthropic’s advice is to grade what was produced. For a task with several parts, keep partial credit as its own result. A support flow that finds the customer and then fails the refund isn't the same event as a flow that dies on the first call.

[Agent task completion rate](https://www.agentrail.co/blog/agent-task-completion-rate) is the full definition of that rate. Use this page for where the rate sits in the set. Use that page when the rate itself is the question. On production traffic, we reconstruct the task agents already attempted, then show where it broke. We do not treat the agent’s last message as the measurement.

## The dead step is its own metric

End-to-end success collapses a failure on step one and a failure on step nine into the same zero. NVIDIA’s step-level score exists so you can see which row in the trace was invalid, redundant, or the one that recovered. On your product, the useful version is concrete. Name the surface and the action. Docs fetch. SDK install. MCP tool. API call. Auth challenge. Payment.

The path metrics around that point are the ones operators already feel. Agents hesitate. They retry. They take steps the task did not need. They fail outright. Count those on completed tasks too. A task that finishes in fifteen calls and two retries is a different experience from a task that finishes in four, even when both “pass.”

[Agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization) is the page that says what to change once you can name that step. The number’s job ends at the name.

## Propensity and efficacy answer two questions

Microsoft’s 6 October 2026 definition splits the choice from the use. Propensity asks whether the agent finds your technology and chooses it when the task does not name you. Their example is a prompt like “add email sending,” with no vendor in the sentence. Efficacy asks whether, once the task names you, the agent uses the current API rather than one you deprecated ([What is Agent Experience (AX)?](https://developer.microsoft.com/blog/what-is-agent-experience-ax/)). A product can be chosen often and used badly. A product can be used well by the few agents that find it. One rate hides that split.

You won't always see propensity in production traffic. Traffic is full of tasks that already pointed at you, plus tasks that never considered you and left no log. A task you write, on a stated agent profile, is how you test the unnamed case. That is a simulation. We run simulations separately from journey reconstruction. The free tier includes 10 simulation runs a month, then $1 a run. Most teams keep both, because a scripted task and a live journey fail in different places.

Efficacy shows up in both places. In traffic, you see whether the agents that already chose you called the current route. In a simulation, you hold the model, the harness, and the prompt still, and you change one thing.

## One pass rate is a point estimate

Agent runs vary when the prompt does not. Anthropic treats each attempt as a trial and asks you to run more than one. Two summaries help. pass@k is the chance of at least one success in k tries. pass^k is the chance that all k tries succeed. Their arithmetic example: a 75% per-trial rate across 3 trials gives (0.75)³, about 42%, for pass^k. That 42% is the example, not a field study. It matters for any workflow a customer will run more than once. A retry loop can post a flattering “eventually” number and still be a poor dependency.

NVIDIA asks you to publish the range across 3 to 5 trials, not the average of a lucky run and a bad one. A spread of 90% and 74% isn't “about 84%.” It's a spread. Microsoft’s practical version, on the same October essay, is to run the same task five times with and without the change. Their glossary line is the scope rule. A result belongs to the profile it was measured with. The profile is the operating system, the harness, the model and its settings, and the extensions loaded.

Anthropic also notes that you can start smaller than a benchmark lab. Twenty to fifty tasks drawn from real failures are enough to see large effects. Wait until the agent is mature before you need a huge suite to detect a small one.

## Path cost sits next to the outcome

Steps, time, and money move even when the pass bit does not. NVIDIA tracks steps per success and cost per success so a shorter, cheaper win is visible next to a noisy one. Token totals alone mislead. Microsoft’s SharePoint Framework upgrades are the cautionary measurement. Claude Sonnet 5 launched with 33% lower per-token pricing than Sonnet 4.6. On those upgrades, in GitHub Copilot Chat, it cost 3.7 times more per run. The sample is 3 scenarios and 15 runs per model. They say the result holds for the scenarios and the profile they measured. Yours can differ. The pattern is the part to copy. Report task cost beside quality, every time.

Human help belongs in the same column. A workflow that completes only after a person steps in twice has been supervised. It has not been delegated. Count interventions, and note why each one happened, when you are running a scripted evaluation. On production journeys, the cousins we can see without a lab are the retries, the extra steps, and the step where the task stopped.

## The record has to hold the denominator

Almost every metric above is a query over a record. If the record is thin, the argument later is about missing fields, not about the product.

For a production journey, keep a stable task identity, the ordered calls and their outcomes, and a terminal state. Keep retries as history. A retry with a recorded trigger is different from a person clicking again. For a scripted run, also store the agent profile. Without it, two runs cannot be compared. Keep refused apart from failed. A correct refusal and a broken workflow are different events. A metric that merges them will punish the safe behavior. The October measurement essay uses four terminal labels for this reason: success, partial, refused, and unsafe.

Redact at write time. Store what someone debugging the dead step would need. A longer log is not automatically a better one.

## Frequently asked questions

### What are agent experience metrics?

Agent experience metrics are task completion, terminal state, the dead step, retries and extra steps, propensity, efficacy, repeatability, and path cost. Read them together. They are not weights in a single score.

### Can we measure this from production traffic alone?

Production journeys show tasks agents already tried, the path they took, and where that path stopped. Propensity, pass^k, and token cost need tasks you script and repeat on a recorded profile. We reconstruct the first group from traffic. Simulations cover the second. [Agent analytics](https://www.agentrail.co/blog/agent-analytics) is the home for the traffic side.

### What is pass^k?

pass^k is the probability that every one of k trials succeeds. Anthropic’s illustration takes a 75% per-trial success rate and three trials, which comes out near 42%. Use it when a customer needs the task to work each time, not on a best-of run.

### Is a readiness score one of these metrics?

A readiness score records a public scan of declared surfaces. It is the precondition described in [what is agent readiness](https://www.agentrail.co/blog/what-is-agent-readiness). It does not say whether a task finished, and it does not belong inside this set as a hidden weight.

## Count the task, then decide what to change

Pick the question before you pick the chart. If the question is whether work finished, use task completion and keep partials separate. If the question is where it died, name the surface and the step, then hand that name to [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization). If the question is whether agents choose you when nobody typed your name, script that task and repeat it.

Agentrail’s analytics surface is built for the production half of this set: the task, the path, the break, and the fix. The free tier includes 100 journeys for an organization each month, then $0.10 a journey. Pricing for journeys, simulations, and the included readiness scan is on the [pricing page](https://www.agentrail.co/pricing).