---
title: "A lab pass is not this week's completion rate"
description: "A simulation grades an agent you configure. This week's completion rate grades production traffic. Keep the two numbers on different charts."
canonical: "https://www.agentrail.co/blog/agent-simulation-vs-production-analytics"
author: "Vivek Mittal"
published: "2026-11-12T12:17:00Z"
last_updated: "2026-11-16T06:16:55.154Z"
---

# A lab pass is not this week's completion rate

A simulation grades an agent you configure. This week's completion rate grades production traffic. Keep the two numbers on different charts.

![A white flask labeled TEST beside a chart labeled REAL](https://images.ctfassets.net/1htfyfnn1o5y/5n9f1S51BOo4tmGRWX8tVb/648e7d46297a6ed296eb539bbfeaa7fd/50ea0069-c681-40f1-82df-807f51c1c826.png?w=1600&fm=webp&q=72)

A simulation grades an agent you configure, on a task you wrote. This week's completion rate grades tasks that agents already attempted in production traffic. Keep the two numbers on different charts. They can disagree on the same day, and both can be right. We don't publish either rate. If a cell has no measurement, leave it empty.

We sell both instruments. Simulations test agents you configure. Journeys reconstruct agents already in production traffic. Most teams use both. The failure is a chart that adds the two numerators and keeps one title.

> **Key Takeaways**
>
> A simulation pass is a result for the agent, the script, and the build you ran.
>
> This week's completion rate covers tasks that showed up in traffic, including tasks you never scripted.
>
> Put the two series on different charts. Adding the numerators changes the population and keeps the old title.
>
> Leave a cell blank until that population is measured.

## A simulation grades an agent you configure

A simulation is a run you start. You pick the agent, the model, and the task, and you decide what counts as a pass for that script. You can run the same pair again when you want a result you can compare.

The agent in that run is the one you selected. Change the model or the task text and the pass belongs to a different simulation.

Our simulation runs put coding agents in sandboxes against preference and experience prompts. Read the pass as a statement about the script, the agent, and the build under test. It says whether this agent, on this task, reached the outcome you wrote. With the agent and the task held fixed, the same setup says whether a docs preview or an SDK preview changed that outcome. Repeats say whether a prompt you control stays stable.

A job that never entered the script can still fail on your API. The sandbox has no record of that attempt.

## The week grades tasks that arrived in traffic

The week's completion rate is a property of traffic. Agents you don't configure arrive with tasks you didn't write. They move across the website, the docs, the API, the SDK, and MCP in the order the task requires. Journeys reconstruct that traffic.

The production record of those paths is [agent journey analytics](https://www.agentrail.co/blog/agent-journey-analytics). The finished share of the week is counted in [agent task completion rate](https://www.agentrail.co/blog/agent-task-completion-rate).

A scripted run you launched at 16:00 has a different denominator from a task an outside agent started. Add the scripted run to the week and the chart looks busier than the traffic was. It often looks more successful, because you chose a task the harness can finish.

The week also contains work the suite never covers. An agent that retries a retired path, or stops on an auth wall, may never appear in the sandbox. That attempt belongs on the production chart. Running the suite again will not create it.

## Keep each series on its own chart

Put the population in the chart title, next to the word "pass."

|  | **Simulation chart** | **Production chart** |
| --- | --- | --- |
| Question | Did this configured agent pass this script? | What did live agents finish in this window? |
| Agent | The one you selected | The ones that arrived |
| Task | The string you wrote | The tasks present in traffic |
| Denominator | Runs of that script | Tasks that started in traffic |
| Numerator | Runs that met the script's outcome | Tasks that met the production outcome |
| Time label | The run, or the build you tested | The week, or the window you chose |

A reader should name the population behind a point without a footnote.

When a review needs both numbers, put them in adjacent columns with different names. Use "Simulation passes, task create-api-key, agent \[name\], build \[id\]." Use "Production completion, week of \[dates\]." Leave each cell blank until that cell is measured.

Write the production numerator from the production outcome. If a deliberate handback is the finish you shipped, that handback is what the week counts. A simulation that scores only a fully unattended success is grading a different finish. Leave that score on the simulation chart.

A public benchmark is a third population. [WebArena](https://arxiv.org/abs/2307.13854) reports a best GPT-4-based agent at 14.41% end-to-end success, against human performance of 78.24% on those tasks. That pair is a result on those tasks.

## Open the chart that matches the question

Open a simulation when you're about to change a prompt, a doc, an SDK, or a tool description, and you can name the task. You need a configured agent and a script so the next run is comparable. The result stays on the simulation chart.

Open the production chart when the question is the week: what agents tried, where they stopped, and whether finished work moved. Another run of a favorite script changes the lab series. Tuesday's traffic is already on the production chart.

Use both when you ship a fix. The simulation asks whether the script you care about improved under an agent you chose. The next week's journeys ask whether traffic moved. Those answers can differ. Traffic may not contain your script. Your script may not contain the path traffic takes.

The change you make after either chart shows a stop is [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization). Know which chart moved before you edit.

## Name the population in the title

Check four edits before you share a chart.

Appending simulation passes to the week's completed tasks adds runs no outside agent started. The rate moves because the population changed.

Replacing the week with the latest lab score, while the title still says production, swaps the points and keeps the label.

Plotting "agent success" with no population label makes a suite run look like a traffic jump. The jump is the suite.

Importing a benchmark percentage as a target line puts someone else's grader on your axis. Put the population in the title, then share the chart.
