--- title: "Test your product, not the agent · Agentrail" description: "Agentrail runs real agents through the jobs in your docs, across models and harnesses, checks the result inside your product, and shows exactly what’s missing where smaller models fail." canonical: "https://www.agentrail.co/use-cases/agent-evals" --- Agent evals for your product # Test your product, not the agent. Agentrail runs real agents through the jobs in your docs, across models and harnesses, checks the result inside your product, and shows exactly what’s missing where smaller models fail. [Check your site’s agent readiness](https://www.agentrail.co/agent-readiness/checker) ![A raccoon in a hoodie running, headphones around its neck and a backpack on.](https://www.agentrail.co/assets/use-cases/agent-evals/hero-raccoon.png) Chasing down every agent failure..... Model lift 96% Task success, Haiku to Opus. Same workflow and harness. Example data. How we check “Done” Every result is checked against your product, never the agent’s word. How it works ## Docs, then a workflow, then a real run, then a check. ### Workflows from your docs Each workflow has a start state, a “done when”, and checks. You can add your own. ### Scenarios agents get asked Some name your product, some don’t. If the agent picks another tool, the workflow isn’t reached. ### Real agents in sandboxes Same scenarios, same harness, only the model changes. Then a second harness such as Codex. ### Checks inside your product A check reads your API, CLI, or account. Never the agent’s own summary. Checks, not claims ## Said done, _wasn’t._ The agent’s last message and the check can disagree. When they do, the run failed, whatever the agent said. ### What the agent said The last message in the run. It’s a claim, so it’s logged but never scored. ### What the product showed Agentrail asks your API what actually happened. When the two disagree, the run failed. Where smaller models stop ## Four kinds of gap. Each one is a product fix. Illustrative, from a sample code review product. Same agent tool, same scenarios: “run a review from the CLI before pushing”. **96%**Opus 5.5 · finished **33%**Haiku 5.5 · finished | Gap | Opus 5.5 | Haiku 5.5 | Fix | | ------------------------------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------- | | Missing flagNo --yesreview --staged asks “Post findings? (y/N)”. | FinishedPiped “y” into the command. | StoppedWaited at the question until the run timed out. | Add a --yes flag. | | Missing docs\--json isn’t written downThe flag works, but no page mentions it. | FinishedTried --json from memory. It worked. | StoppedRead the coloured table and missed two findings. | Document --json. | | Unclear errorNo schema for .review.ymlNo published schema for team rules. | Not reportedNo result. | StoppedMade up field names, gave up after three refusals. | Publish the schema. | | Needs a personA link only a person can clickSign-up confirmation and the licence download. | FinishedAsked the user to click it. | StoppedKept calling /v1/me until the run timed out. | Remove the human-only step.44 of 96 runs needed a person. | Steps and tokens ## The shortest way, next to a real run. Every extra step costs your customers tokens and time. Agentrail shows which ones were yours to fix. - **Your product**Steps after an error, a docs gap, or an unanswered question. - **The agent**Wandering through files, or repeating a command that worked. - **Reading**Tokens of docs, CLI output and API answers per finished run. Six steps to done. Not fourteen..... ![A raccoon climbing a short flight of stairs toward a checkered flag.](https://www.agentrail.co/assets/use-cases/agent-evals/steps-raccoon.png) Safe access ## Test credentials never go in a prompt. Agents need a way in. Agentrail hands each credential to the run when it starts, and keeps it out of the run’s record. - **Test account**A login made only for agent runs. - **GitHub test org**A sandbox organization, never your real one. - **Throwaway inbox**For self-sign-up and email confirmation. A run that leaks one fails its safety check. Test access Handed to each run when it starts. Never in the prompt. - Test account agent-test@review-eval.dev Account and API key - GitHub test organization review-eval GitHub App installation - Agents sign up themselvesPer run A throwaway inbox per run - SSO test tenant Okta developer tenant - Self-hosted licence A file mounted into the run Kept out of the run’s record MCP servers and agentic tools ## Does your MCP server work the same in every agent? A tool that works in Claude Code on Opus can fail in Codex, or on a smaller model. Consumer AI companies use the same runs to test their own tools. - **Works everywhere**Connecting the MCP server finishes on every model. Nothing in it leans on a smart agent. - **Gets in the way**list\_findings returns 40 KB for a large pull request. One model copes, another doesn’t. Finish rate By workflow and model · illustrative 0%100% HaikuSonnetOpusCodex Connect MCP server 88%92%96%92% Run a review 33%61%96%70% List findings_40 KB_ 40%74%94%52% Post findings 58%80%95%88% Team rules 25%55%90%60% Self-host setup 20%45%85%50% **1 of 6**workflows works on every model. Darker means more runs finished. Start free ## Agent readiness checks whether agents can find and read. A free scan of your public site, docs and API, scored out of 100, with the change worth the most named under “Fix next.” Readiness checks whether agents can find and read things. A simulation checks whether they finish. [Check your site’s agent readiness](https://www.agentrail.co/agent-readiness/checker) ![A raccoon in a detective coat, holding a magnifying glass.](https://www.agentrail.co/assets/use-cases/agent-evals/banner-raccoon.png) Fix and run again ## The output is a list of product fixes. Every result ends in a change you can make, a pull request you approve, and a re-run that shows it worked. 1. 01 ### A recommendation, with the evidence Add --yes and document --json. Two small changes, tied to the runs that stopped. 2. 02 ### A pull request you review Coon, the in-app assistant, raises it. Nothing merges without you. 3. 03 ### The same scenarios, again Agentrail re-runs the workflow and shows what changed: finish rate, model gap, and where runs still stop. ![A ranked list of product fixes, each with the evidence behind it and a pull request to raise.](https://www.agentrail.co/assets/features/4.webp) FAQ ## Questions teams usually ask. How is this different from agent eval tools? The agent’s last message is logged and never scored. A check reads your API, CLI, or account. When the two disagree, the run failed, whatever the agent said. Which agents do you run? Real agents in sandboxes. The same scenario runs on one harness across models, then on a second harness such as Codex. Only the model or the harness changes. Can you test our MCP server or agent tools? Yes. The same runs check whether connecting the server, listing findings, and posting results finish on every model and harness. A tool that works in Claude Code on Opus can fail in Codex, or on a smaller model. Where do workflows come from? From the jobs in your docs. Each workflow has a start state, a “done when”, and checks. You can add your own. Some scenarios name your product, and some don’t. How do you know the agent really finished? A check reads your product: an API response, a CLI command, or the state of an account. “All done” in the transcript is not a pass. Do you need our credentials? Not production ones. Each run gets a test account, a sandbox organization, or a throwaway inbox when it starts. Those stay out of the prompt and out of the run’s record. What’s the difference between Agent readiness and a simulation? Readiness checks whether agents can find and read your public site, docs, and API. A simulation checks whether they finish the job.