---
title: "Where This Task Died, and What Changed the Pass Rate"
description: "One agent task, the step where it stopped, the change you made, and the pass rate on a fair rerun. A method teardown with fields ready for real customer data."
canonical: "https://www.agentrail.co/blog/agent-task-pass-rate"
author: "Harsh Raj"
published: "2026-11-02T00:00:00Z"
last_updated: "2026-11-06T06:26:51.880Z"
---

# Where This Task Died, and What Changed the Pass Rate

One agent task, the step where it stopped, the change you made, and the pass rate on a fair rerun. A method teardown with fields ready for real customer data.

![Three task cards on a dark blue gradient: a failed document, a gauge, and a passed checklist](https://images.ctfassets.net/1htfyfnn1o5y/4higFfpN1DtsBp11YBiJls/09a5f88239d2d2d30f9f70fb83941ee5/80fce8a3-6d69-4e9b-bd25-fdb22710d0a6.png?w=1600&fm=webp&q=72)

An agent task pass rate only moves when you can point at the step that failed. This page is a method case study for that sequence: one task, the surface where it stopped, the change you shipped, and the pass rate on a fair rerun.

It is written as a template until a named customer proof is public. Fields in `[brackets]` are waiting for real data. Do not treat bracketed labels as Agentrail results. The definition of the underlying fraction is [agent task completion rate](https://www.agentrail.co/blog/agent-task-completion-rate). Diagnosis classes live in [why AI agents fail tasks](https://www.agentrail.co/blog/why-ai-agents-fail-tasks). The fix playbook lives in [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization).

> **Key Takeaways**
>
> • Write the task string and the environment outcome rule before you watch a run.
>
> • Name the dead step as surface plus action, not as “the agent struggled.”
>
> • Change one surface that matches that step, then rerun the same harness and the same success rule.
>
> • Report `[before pass rate]` and `[after pass rate]` over `[n]` trials. Leave the cells blank until measured.
>
> • Public method cousins (Anthropic, Cloudflare, academic prototypes) show that scoped changes can move outcomes. They are not Agentrail customer proofs.

## Write the task and the outcome rule before you watch a run

Pass rate is completed trials divided by started trials for one named task. The unit is the task. The outcome is a state in the environment, not the agent’s closing sentence. Anthropic’s January 9, 2026 note on agent evals separates the transcript from the outcome: the agent can say a flight is booked while the database has no reservation ([Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)).

Fill these lines first:

> `Customer: [Customer]`
>
> `Task: [task string an agent is given]`
>
> `Success criteria: [checkable state: key exists, PR opened, record matches schema]`
>
> `Failure criteria: [what counts as fail, including false completion]`
>
> `Harness: [agent product + model + version]`
>
> `Start rule: [first relevant action on our docs / API / SDK / MCP]`

If you cannot write the success criteria without reading the agent’s last message, you are not ready to publish a pass rate. Use the definition page for the fraction itself. This page owns the teardown around it.

## Name the step where the task died

A useful failure report names a place. Docs returned a directory listing. The OpenAPI example used a retired path. The MCP tool returned a stack trace. Auth answered 401 with an HTML login page. The agent retried the same call until the context filled.

Template line:

> On `[date]`, `[Customer]`’s task “`[task]`” stopped at `[surface]`: `[action]`. Evidence: `[request / tool result / screenshot of state]`.

[Why AI agents fail tasks](https://www.agentrail.co/blog/why-ai-agents-fail-tasks) groups the common places a task dies (missing doc, unusable error, auth wall). Use that taxonomy to classify the stop. Do not open a sitewide checklist until the class is written down.

Write the stop in language a second engineer can reproduce. “Auth was confusing” is not a step. “POST /oauth/token returned HTML from the login page with status 401” is a step. The second form tells you whether the next edit belongs in docs, in the API error body, or in the redirect rules.

If the agent never reached your product, the dead step may be discoverability or an unaided preference miss. That is still a valid stop. Measure it with a preference test first, then return here when an agent that chose you still fails the task. [tools AI coding agents recommend](https://www.agentrail.co/blog/tools-ai-coding-agents-recommend) owns named vs missing. This page owns pass rate after the agent is already trying.

## Match one change to that step

Optimization starts at the dead step. A docs rewrite, a clearer JSON error, a stable OAuth path, and a tighter tool description are different edits. [Agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization) is the playbook for matching the edit to the failure.

Record the change as a single sentence:

> Change shipped `[date]`: `[PR / docs diff / MCP description / error body]`. Intent: fix `[dead step]`. Out of scope: `[everything else you did not change]`.

If you change three surfaces at once, the after rate cannot teach you which edit mattered. Prefer one change, then a rerun, then the next change.

## Rerun with the same harness and the same success rule

A fair rerun keeps the task string, the harness profile, and the outcome check fixed. Only the product surface under test should differ.

Report:

> `Trials before change: [n]`
>
> `Passes before: [k]`
>
> `Pass rate before: [before %] = k / n`
>
> `Trials after change: [n]  (same n unless you document why not)`
>
> `Passes after: [k2]`
>
> `Pass rate after: [after %] = k2 / n`

If you use best-of-n sampling, say so. Anthropic’s pass@k vs pass^k distinction exists because one lucky trial is not reliability. For a customer teardown, prefer the same trial count before and after, scored with the same environment check.

Exclude false completion from the numerator. If the agent said the key was created and the environment has no key, that trial is a fail for pass rate. You may track eloquence as a separate quality note. Do not bury it inside `[after %]`.

Partial credit belongs beside the rate, not averaged into it. If the agent authenticated but never created the resource, report that class as its own count. The definition page warns against inventing weights for partial states. Keep the same discipline here.

Journeys are how Agentrail reconstructs the path across website, docs, API, SDK, and MCP when the task is already in traffic: 100 free per organization per month, then $0.10 per journey ([pricing](https://www.agentrail.co/pricing)). Simulations remain the scripted arm when you need a sandbox rerun. Price is not a pass rate.

Production traffic and scripted reruns answer different questions. A journey shows where live agents stopped. A simulation shows whether your fix holds under a prompt you control. Most teams need both. This teardown works for either instrument if the success rule stays fixed.

## Before and after

| Field | Value |
| --- | --- |
| Customer | `[Customer]` |
| Task | `[task]` |
| Dead step | `[surface]: [action]` |
| Change | `[one-line diff]` |
| Pass rate before | `[before %]` over `[n]` |
| Pass rate after | `[after %]` over `[n]` |
| Window | `[dates]` |
| Instrument | `[journeys / simulations / lab harness]` |

Until those cells are measured for a named customer, leave them as brackets. Publishing a method without a fake percentage is preferable to inventing proof.

## Public method cousins, not our customer proof

Scoped changes do move agent outcomes in public write-ups. None of the figures below are Agentrail journey results. Use them to justify running the method, not to fill `[after %]`.

Anthropic’s June 13, 2025 research-system note describes a tool-testing loop that rewrote a flawed MCP tool description. Later agents using the new description finished tasks in 40% less time because they skipped most of the earlier mistakes ([How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system)). That is one internal loop on their tools.

Cloudflare’s April 17, 2026 Agent Readiness post reports that after they refined docs maps (`llms.txt` splits, fewer empty directory pages), one agent on their docs used 31% fewer tokens and reached the correct answer 66% faster than on the average site that had not been refined the same way ([Introducing the Agent Readiness score](https://blog.cloudflare.com/agent-readiness/)). That is Cloudflare’s comparison on Cloudflare’s docs.

Elnaffar and Rashidi’s July 13, 2026 preprint held models fixed and changed page structure on a prototype shop. Strict success moved from 74 of 150 runs (49.3%) on the human-oriented baseline to 134 of 150 (89.3%) on the agent-ready version ([arXiv:2607.12056](https://arxiv.org/abs/2607.12056)). It is a prototype study, not a forecast for your domain.

These cousins share a shape with this teardown: name the failure, change the surface, measure the same task again. Fill `[Customer]` when you have a real one.

What they do not share is your denominator. Cloudflare’s docs comparison is not your signup task. Anthropic’s tool-description loop is not your MCP server. The preprint shop is not your production API. Cite them as proof that scoped edits can move outcomes. Then measure your own `[before %]` and `[after %]`.

## What not to put in the after column

Do not paste a readiness grade into the pass-rate table. Do not paste a preference mention rate either. Those instruments matter. They are not this rate.

Do not invent a customer name to make the template look finished. A bracketed `[Customer]` in a public method piece is clearer than a fake logo row.

Do not change the success rule between arms to make the after number kinder. If the before arm required a working key, the after arm requires a working key. If you broaden the outcome, say you broadened it, and treat that as a new experiment.

## Frequently asked questions

### What is agent task pass rate in this article?

It is the share of trials of one named task that reach the required environment state, under a fixed harness and start rule. The formal definition of the production fraction is on [agent task completion rate](https://www.agentrail.co/blog/agent-task-completion-rate).

### Do we need a named customer to publish the method?

No. A public method piece with brackets is honest. A named customer with invented rates is not. Prefer empty cells over fiction.

### Can a readiness score replace this teardown?

No. A readiness scan is a public precondition. Pass rate is whether a task finished. A mid-board score can sit beside a failed task. Keep the instruments separate.

### Where do we see the path across docs, API, and MCP?

[Agent analytics](https://www.agentrail.co/blog/agent-analytics) is the category. Journeys stitch one task across those surfaces. Start from the journeys line on [pricing](https://www.agentrail.co/pricing).

## Fill the brackets with one real task

Pick one workflow that already fails for agents. Write the outcome rule. Capture the dead step. Ship one change. Rerun. Put the numbers in the table only when they exist. Diagnosis stays in [why AI agents fail tasks](https://www.agentrail.co/blog/why-ai-agents-fail-tasks). Fixes stay in [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization). When you want the production path reconstructed, journeys are on [pricing](https://www.agentrail.co/pricing).