Skip to main content
Developers increasingly ask a coding agent which product to use. Agent Preference runs the requests your buyers would type, and shows how often agents reach for your product instead of a competitor’s, and why.

​
How it works

You write preference prompts: requests a developer would type to a coding agent. Each time a prompt runs, one agent with one model answers it on its own, and Agentrail records which products it found, named, recommended and finally chose. The Agent Preference page adds those runs up. A prompt has one of two types: In the results, prompts fall into three prompt types:
  • Organic: open-ended prompts that do not name your product.
  • Branded: open-ended prompts whose text names your product, or one of its Aliases from the Brand Vault.
  • Head-to-head: head-to-head prompts.
Results cover your own product and the products on your Competitors page. A product an agent picks that is on neither list shows as Not listed.
Agentrail recognises products by the names you give them. Add the other names people use: Aliases in the Brand Vault, Other names on a competitor.

​
Add a prompt

Only an Admin can add, edit, run or delete prompts.
1

Open the prompt list

Open Agent Preference in the sidebar and scroll to Prompts.
2

Start a new prompt

Click Add prompt. The Add preference prompt dialog opens.
3

Fill in the prompt

Choose a Type, then write the Prompt the way a developer would ask a coding agent. For a head-to-head prompt, enter the Alternative: the other product the prompt names.
4

Group it

Choose a Topic, and any Tags. See Topics and tags.
5

Choose how it runs

Pick a Repository for runs to start from, or leave it on Empty project. Tick the Agents and models to run it with. See Executions.
6

Save

Click Add prompt.
A prompt without a topic is flagged as needing attention and cannot run until you give it one. To list those prompts, turn on Needs attention above the list. A head-to-head prompt with no alternative shows No alternative; edit it to add one. To change or remove a prompt, open the actions menu on its row and choose Edit or Delete. Deleting removes the prompt from every list, along with its tag assignments. Editing changes future runs only: each run keeps the text it ran with.

​
Run a prompt

Open a prompt’s actions menu and choose Run, or open the prompt and click Run. Agentrail starts one run for each agent and model the prompt has, and says how many it started. Prompts also run on their own, on the schedule set in Run settings. See Executions for both.

​
Read the results

The top of the Agent Preference page sums up runs in the chosen view:
  • Prompt type: All prompt types, Organic, Branded or Head-to-head.
  • Period: Last 7 days, Last 30 days (the default), Last 90 days, Last year or All time, by when runs ended.
  • Agent and model (All agents) and tag (All tags).
  • A tab for each topic, after All topics.
Each figure is a percentage with the runs behind it, such as “10 of 16 runs”. When nothing can be counted it says No measurable results, never 0%.
Every product by how often agents Recommended it, with how often it was Chosen. Click a product to open its page.
Your product against each alternative your head-to-head prompts name: wins and losses, and whether you are Winning, Tied or Losing. The neither count is runs where the agent picked neither product, or a third one. Show head-to-head prompts lists the prompts behind it.
How often agents mentioned, recommended and chose your product, per day, week or month depending on the period.
Four steps for each product: Discovered (it turned up in a web search or a page the agent read), Mentioned (the agent named it), Recommended, and Chosen (it was the agent’s final choice). Each step counts out of every run, with the share of the step before that reached it.
Every final choice, including products nobody listed and runs with no choice: None: asked a question, None: listed options without choosing and None: no answer.
Runs where an agent gave a reason not to use a product, in its own words. Step links open the run’s log where it said so.
Per agent and model, how often it Answered from memory: without searching or reading a page first.
Of the runs whose final choice was your product or a competitor, how often the code uses that product. Filter to prompts that ask for code to make this meaningful.

​
A prompt’s page

Click a prompt in the list to open it. It shows the prompt’s settings, Results for this prompt over the last 30 days across every agent, and every run of it under Executions.

​
A product’s page

Click a product name to see its results on its own, the reasons agents gave against it, and Runs that mentioned it, each with the step it was first named.

​
Runs that are not counted

Results count runs that Finished, Failed or Timed out. When a run is left out, the figure says why under Not counted:
  • not judged yet: the run has just ended. Results arrive a few minutes after a run ends.
  • judging failed: Agentrail could not work out this run’s result.
  • judged with an earlier question: the run will count once it is read again.
  • interrupted: the run stopped for a reason on Agentrail’s side.
A figure may also say how many runs were left out because the agent cannot show that detail, or how many came from an incomplete record.

​
Limits

​
Who can do what

See People and roles.

Agent Experience

Give agents a task with your product and see how far they get.

Executions

Start runs, set the schedule and read each run.