---
title: "How to write the pass rule for one agent task"
description: "Write the pass rule before the run: the outcome, the evidence you will read, and what must not flip the bit."
canonical: "https://www.agentrail.co/blog/agent-task-success-criteria?h=5CYJE1JNYT2QS00Z"
author: "Sidhant Goyal"
published: "2026-11-17T12:19+00:00"
last_updated: "2026-11-17T11:06:45.150Z"
---

# How to write the pass rule for one agent task

Write the pass rule before the run: the outcome, the evidence you will read, and what must not flip the bit.

![A white path of connected nodes on a blue gradient](https://images.ctfassets.net/1htfyfnn1o5y/4ahkbPp3CvyhSjCrXQaeyy/f80ff5a9ea5966adf6658eea403cf18c/d006a191-80f5-4f70-b011-18436284fd4a.png?w=1600&fm=webp&q=72)

A pass rule is the check that decides whether one agent task finished. Write it before the run as three lines: the outcome, the evidence, and the ungraded list. The fraction those bits become is counted in [agent task completion rate](https://www.agentrail.co/blog/agent-task-completion-rate?h=5CYJE1JNYT2QS01X).

When the outcome is a write, the evidence is a later read of product state. That read is [reading the record back after a write](https://www.agentrail.co/blog/verify-an-api-write-with-an-ai-agent?h=5CYJE1JNYT2QS02V).

> **Key Takeaways**
>
> The outcome is a state the product can hold, outside the agent's last message.
>
> The evidence is the read you will perform. An errored read stays out of the pass column.
>
> Ungraded notes can be recorded. They stay off the pass bit.
>
> Write the card before the run, for one task only.

## Write the outcome as a state the product can hold

The outcome is the state that must be true when the task has finished. It names a record, a file, a credential, or a resource, plus the condition on it. A second engineer should be able to check the sentence without opening the transcript.

These are shapes, not results from a customer:

- The API key exists, and a follow-up read with that key returns the caller's account.
- The pull request is open on the named repository, and the diff includes the file the task named.
- The customer record exists, and `status` is the value the task required.
- The handback record exists and names the task, when a human step is the designed end.

The agent's last message, a confident tone, the word "success" in the trace, and a low token count are comments. If your sentence quotes the agent's wording, rewrite it until the nouns are things the product can store.

If the designed end is a handback, write that handback into the outcome. A card that passes only a fully unattended finish will fail a product that is built to stop for a person.

## Name the read that proves the outcome

Evidence is the observation that proves the outcome. It is a read of product state: a GET, a query, a file open, or a command whose stdout you parse. One primary read is enough. Add a second read only when the outcome has a second condition. Each read maps to a condition you already wrote.

The evidence line names where you read, which fields matter, and what you do on a bad read. A bad read is an error, a timeout, or an identity other than the one the task named.

An errored read is an unreadable check. Fix the evidence path, or mark the trial unread and keep it out of the pass column. Do not fill the hole with the agent's summary. Unread is a status. It carries no partial credit.

The attempt you are grading may already be a production record. That record is [agent journey analytics](https://www.agentrail.co/blog/agent-journey-analytics?h=5CYJE1JNYT2QS03S). Use the same outcome and the same evidence on a reconstructed journey.

## List what must not flip the bit

Ungraded means you may record it, and it must not change the pass bit. Unless the task itself is about them, keep the closing summary, tone, length, token count, and code style off the bit. Latency stays off the bit unless the task is a latency target.

Partial progress stays beside the bit. Note the last true condition. Award a pass for an intermediate state only when the outcome says that state is a full pass.

A clearer error can change whether later trials pass. That correction is a product edit in [agent experience optimization](https://www.agentrail.co/blog/agent-experience-optimization?h=5CYJE1JNYT2QS04Q), scored on later trials.

Give the ungraded items their own columns, so a messy pass and a tidy failure both stay visible.

## Write one card before either run

A pass rule written after you read the transcript will fit the runs you already liked. Write this card first.

`Task: [one task string]`

`Outcome: [state outside the transcript]`

`Evidence: [read, fields, identity]`

`Pass: evidence shows the outcome`

`Fail: evidence missing, mismatched, or a different record`

`Unread: [what you do if the read errors]`

`Ungraded: [summary, tone, tokens, style, partial progress]`

One card, one task. "Create a key" and "rotate a key" need two cards, even though both mention keys. Write both before either run. If only one model can satisfy a clause, that clause is a transcript habit.

Leave the next two blocks as shapes until you have a real task string. Do not attach a rate to either one.

`Task: create a webhook`

`Outcome: a webhook on the named account whose url equals the task URL`

`Evidence: GET the webhook by id; compare url`

`Fail: 404, or url differs`

`Ungraded: the POST status text, the agent's explanation`

`Task: open a pull request that fixes the install block`

`Outcome: a PR on the named repo whose diff contains the required command`

`Evidence: read the PR and the file patch`

`Ungraded: the PR description`

## Keep a missing read out of the pass column

Fail open means a trial passes when the outcome is false. The usual cause is evidence that trusts the transcript, the HTTP status of a write, or a log line the agent printed.

Fail closed means a trial fails when the outcome is true. The usual cause is an extra clause the state does not need. You required a phrase in the summary, a field the task never named, or a second resource that was not the job. The record is right, and the card still says no.

Read the trials you already stored and decide which error you will keep. Pass the trial when the evidence shows the outcome. Keep a missing or unreadable read out of the pass column. The same card can grade a sandbox run and a live journey. Keep those counts on their own charts.

Name an owner on the card. A change applies to later trials. Trials already scored keep the card they were scored with, and the date of the change goes in the write-up. Put the outcome, the evidence, and the ungraded list on the card, then start the run.
