> ## Documentation Index
> Fetch the complete documentation index at: https://agentrail.co/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# PR checks

> On pull requests to your docs, SDK, CLI or MCP server, run the agent jobs you list against the pull request's real build, before you merge.

You list the jobs you care about in `agentrail.yml` at your repository's root. When a pull request is opened ready for review, marked ready, or re-run, Agentrail runs each listed job against the pull request's preview build and reports on the pull request whether agents still get it done.

## Set it up

PR checks use your workspace's GitHub connection (**Integrations**, then **GitHub**). A GitHub owner approves the app's permissions once, on GitHub: checks, pull requests, deployments and actions, beside read access to contents. The app never writes code.

Then add the check file. Open **PR checks** in Agentrail and pick the repository: Agentrail drafts `agentrail.yml` from your workflows whose runs used that repository's package, docs pages or MCP tools. Copy it into the repository's root, or ask your coding agent to fetch it through Agentrail's MCP server.

```yaml theme={null}
checks:
  - workflow: verify-a-sending-domain
    agent: claude-code
    runs: 2
```

Each entry names one of your workflows by its name (its title in lower case, words joined by `-`), an agent (`claude-code`, `codex`), and how many runs to make. A repository without the file gets a skipped check that points to the draft.

A check also needs:

* **A preview of each commit**, made by your own CI: a preview deployment (for docs or a hosted MCP server) or a preview package on pkg.pr.new (for an npm package). Agentrail never builds anything.
* **Your repository in your Brand Vault**: the package name in `package.json`, or the address your production deployment serves, matches one of your products' packages, docs or MCP servers.

Draft pull requests are not tested. Pull requests from forks are never checked.

## When a check tests

A check tests when a pull request is opened ready for review, when it is marked ready for review, and when you press **Re-run** on the check in GitHub. A later push is not tested: its check run ends skipped, shows what changed, names the commit last tested and its result, and says Re-run tests this commit. A skipped check run satisfies GitHub's required checks.

## What a check does

1. An **Agentrail** check run appears on the pull request, in progress.
2. Agentrail reads `agentrail.yml` at the pull request's commit, so a pull request can change its own checks.
3. It finds the pull request's build and the base build: for a package, the build of the commit your pull request branched from; for docs or a hosted MCP server, your live site.
4. It compares the builds, for free: the files of your package that differ and the methods and types added, removed or changed; the docs pages whose text differs; the MCP tools added, removed or changed. What changed shows on the check run right away. When nothing differs, the check ends skipped with no runs.
5. For each entry, agents run the job `runs` times on the pull request's build.
6. If every counted run got the job done, the entry **passed**. If any did not, the same runs go on the base build: when those get it done, the entry **fails: new in this PR**; when they do not, it is **already failing** on your default branch.

A run counts only when it reached its build (it installed the package, read a changed page, or called the MCP server) and did not end because of Agentrail's side. An entry with no counted run is **not tested**, with the reason. Steps and tokens are shown for each run; they never decide the result.

## Read the result

| Conclusion | When |
| - | - |
| **Failure** | An entry failed on this pull request and passed on the base build |
| **Neutral** | No entry failed because of this pull request, but one is already failing, not tested, or names a workflow or agent your workspace does not have |
| **Success** | Every entry passed |
| **Skipped** | No `agentrail.yml`, nothing agents reach changed, or a later push |

With `runs: 2`, a job that works 3 times in 4 can miss on the pull request by chance and pass on the base by chance. A larger `runs` lowers that chance.

Once a check starts its runs, Agentrail writes one summary comment on the pull request and keeps it up to date for the newest tested commit. Each entry that fails as new in this pull request gets one comment: on the added line that declares the failing method, tool or docs sentence when there is one, otherwise in the summary. On GitHub, agents are named by harness only (Claude Code, Codex), never by model.

A check is cancelled when a Re-run on a newer commit replaces it, when the pull request is closed or merged, or when Agentrail could not finish it: its runs that have not started are cancelled; running runs finish.

Whether a failing check blocks merging is your own GitHub setting.

## In Agentrail

Open **PR checks** under Executions to see every check: its builds, what changed, and a card per entry with its result, its counts on each build, and every run with its steps and tokens. Your agents can read the same, and draft the check file, through Agentrail's MCP server, CLI and Ask Agentrail.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.