> ## Documentation Index
> Fetch the complete documentation index at: https://agentrail.co/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Experience

> Give coding agents a realistic task with your product and see how far they get, what they claim, and what stops them.

When a developer asks a coding agent to build with your product, the agent has to find your docs, install your package, set it up and make it work. Agent Experience runs those tasks for you and shows where agents get stuck, so you know what to fix in your product and its docs.

## How it works

You write **experience evaluations**. Each one is a realistic task for a coding agent plus a checklist of what a correct result looks like:

* **Task**: what a developer would hand the agent. State every requirement the result could fail on.
* **Criteria**: each one something a reviewer can answer yes or no by reading the finished work, in the order they would check them.

Each run gives one agent with one model the task alone, in a starter repository. The agent never sees your criteria.

## Add an evaluation

Only an Admin can add, edit, run or delete evaluations.

<Steps>
  <Step title="Open the evaluation list">
    Open **Agent Experience** in the sidebar and scroll to **Evaluations**.
  </Step>

  <Step title="Start a new evaluation">
    Click **Add evaluation**. The **Add experience evaluation** panel opens.
  </Step>

  <Step title="Write the task">
    Fill in **Task** the way a developer would ask a coding agent.
  </Step>

  <Step title="Group it">
    Choose a **Topic**, and any **Tags**. See [Topics and tags](/docs/guides/topics-and-tags).
  </Step>

  <Step title="Add criteria">
    The first criterion is ready to fill in. Click **Add criterion** for each one after it. Use the arrows to reorder them, and the bin to remove one.
  </Step>

  <Step title="Choose how it runs">
    Pick the **Repository** runs start from. It starts on the default starter repository. Tick the **Agents and models** to run it with. See [Executions](/docs/guides/executions).
  </Step>

  <Step title="Save">
    Click **Add evaluation**.
  </Step>
</Steps>

<Tip>
  Write criteria a person could check without asking the agent anything. "The signup form sends a welcome email" is checkable; "the code is good" is not.
</Tip>

An evaluation without a topic, or without any criteria, is flagged as needing attention and cannot run until you fix it. In the list, **No criteria** marks one with none. To find them all, turn on **Needs attention** above the list.

To change or remove an evaluation, open the actions menu on its row and choose **Edit** or **Delete**. Deleting removes the evaluation from every list, along with its criteria and tag assignments. Editing changes future runs only.

## Run an evaluation

Open an evaluation's actions menu and choose **Run**, or open the evaluation and click **Run**. Agentrail starts one run for each agent and model it has. Evaluations also run on their own, on the schedule set in **Run settings**. See [Executions](/docs/guides/executions).

## Read the results

The top of the **Agent Experience** page sums up runs in the chosen view. Choose a period (**Last 30 days** by default), an agent and model (**All agents**), a tag (**All tags**), or a topic tab. Each figure is a percentage with the runs behind it; **No measurable results** means nothing could be counted.

<AccordionGroup>
  <Accordion title="How far the task got">
    Five steps with your product: **Read the docs**, **Installed**, **Configured**, **Integrated** (the code uses your product) and **Checked** (the agent's own check, such as its tests or build, passed after its last change). Each step counts out of every run, with the share of the step before that reached it.
  </Accordion>

  <Accordion title="Over time">
    How often agents integrated your product and checked their work, per day, week or month depending on the period.
  </Accordion>

  <Accordion title="Claimed and checked">
    What the agent said it did: **Claimed done**, **Claimed partly done**, **Said not done** or **Stopped to ask**. Beside it, **Check passed, of those that claimed done**, and **Claimed done without a passing check**: agents that said they were done without proving it.
  </Accordion>

  <Accordion title="Your product or another">
    Which product the code used: **Your product**, **A competitor** or **Neither**.
  </Accordion>

  <Accordion title="What a person must still do">
    Steps the agent said a person must take before the feature works: **Add an API key**, **Create an account**, **Verify a domain or sender**, **Install an app or integration**, **Configure the dashboard** and **Run a command**. Open the quotes to read the agent's words.
  </Accordion>

  <Accordion title="Failures by kind">
    Failed steps sorted into **Problems in your product** (API or SDK errors, documentation problems, package install failures, version or dependency conflicts, missing credentials or account) and **Everything else**. A run counts once per kind. **Problems, run by run** quotes each one, linked to its step in the run's log.
  </Accordion>

  <Accordion title="Time and tokens">
    Per agent and model: **Runs**, **Average time**, **Average tokens** and **Average cost**.
  </Accordion>
</AccordionGroup>

Results count runs that **Finished**, **Failed** or **Timed out**. Runs left out are listed under **Not counted** with the reason, the same way as on [Agent Preference](/docs/guides/agent-preference#runs-that-are-not-counted).

### An evaluation's page

Click an evaluation's task in the list to open it. It shows the task, its numbered criteria, its settings, **Results for this evaluation** over the last 30 days across every agent, and every run of it under **Executions**.

## Limits

| What | Limit |
| - | - |
| **Task** | 4,000 characters |
| **Criteria** on one evaluation | 50 |
| One criterion | 1,000 characters |
| **Tags** on one evaluation | 20 |

## Who can do what

| Action | Admin | Member |
| - | - | - |
| See results, evaluations and runs | Yes | Yes |
| Add, edit and delete evaluations | Yes | No |
| Run an evaluation | Yes | No |

See [People and roles](/docs/guides/people-and-roles).

<CardGroup cols={2}>
  <Card title="Agent Preference" icon="scale" href="/docs/guides/agent-preference">
    See which products agents recommend and choose.
  </Card>

  <Card title="Executions" icon="history" href="/docs/guides/executions">
    Start runs, set the schedule and read each run.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.