agentrail.yml at your repository’s root. When a pull request is opened ready for review, marked ready, or re-run, Agentrail runs each listed job against the pull request’s preview build and reports on the pull request whether agents still get it done.
Set it up
PR checks use your workspace’s GitHub connection (Integrations, then GitHub). A GitHub owner approves the app’s permissions once, on GitHub: checks, pull requests, deployments and actions, beside read access to contents. The app never writes code. Then add the check file. Open PR checks in Agentrail and pick the repository: Agentrail draftsagentrail.yml from your workflows whose runs used that repository’s package, docs pages or MCP tools. Copy it into the repository’s root, or ask your coding agent to fetch it through Agentrail’s MCP server.
-), an agent (claude-code, codex), and how many runs to make. A repository without the file gets a skipped check that points to the draft.
A check also needs:
- A preview of each commit, made by your own CI: a preview deployment (for docs or a hosted MCP server) or a preview package on pkg.pr.new (for an npm package). Agentrail never builds anything.
- Your repository in your Brand Vault: the package name in
package.json, or the address your production deployment serves, matches one of your products’ packages, docs or MCP servers.
When a check tests
A check tests when a pull request is opened ready for review, when it is marked ready for review, and when you press Re-run on the check in GitHub. A later push is not tested: its check run ends skipped, shows what changed, names the commit last tested and its result, and says Re-run tests this commit. A skipped check run satisfies GitHub’s required checks.What a check does
- An Agentrail check run appears on the pull request, in progress.
- Agentrail reads
agentrail.ymlat the pull request’s commit, so a pull request can change its own checks. - It finds the pull request’s build and the base build: for a package, the build of the commit your pull request branched from; for docs or a hosted MCP server, your live site.
- It compares the builds, for free: the files of your package that differ and the methods and types added, removed or changed; the docs pages whose text differs; the MCP tools added, removed or changed. What changed shows on the check run right away. When nothing differs, the check ends skipped with no runs.
- For each entry, agents run the job
runstimes on the pull request’s build. - If every counted run got the job done, the entry passed. If any did not, the same runs go on the base build: when those get it done, the entry fails: new in this PR; when they do not, it is already failing on your default branch.
Read the result
With
runs: 2, a job that works 3 times in 4 can miss on the pull request by chance and pass on the base by chance. A larger runs lowers that chance.
Once a check starts its runs, Agentrail writes one summary comment on the pull request and keeps it up to date for the newest tested commit. Each entry that fails as new in this pull request gets one comment: on the added line that declares the failing method, tool or docs sentence when there is one, otherwise in the summary. On GitHub, agents are named by harness only (Claude Code, Codex), never by model.
A check is cancelled when a Re-run on a newer commit replaces it, when the pull request is closed or merged, or when Agentrail could not finish it: its runs that have not started are cancelled; running runs finish.
Whether a failing check blocks merging is your own GitHub setting.