---
title: "How to Ask an Agent to Pick a Tool Without Naming Yours"
description: "Unaided first, head-to-head second. A method for AI agent recommendation tests that keeps your brand out of the opening prompt."
canonical: "https://www.agentrail.co/blog/ai-agent-recommendation-test"
author: "Sidhant Goyal"
published: "2026-11-02T00:00:00Z"
last_updated: "2026-11-06T06:26:52.144Z"
---

# How to Ask an Agent to Pick a Tool Without Naming Yours

Write an unaided preference prompt from a real developer job, keep your product name out of the first question, then run a head-to-head AI agent recommendation test.

![A comparison table on a dark navy gradient, with Your tool selected across picked, installed, and works](https://images.ctfassets.net/1htfyfnn1o5y/6IplnucBrn0AQN7qhO9E60/fc33228fdc7ded5c32511c71fa84eb21/d1549d4c-9cc7-4e89-a479-69738298cd4c.png?w=1600&fm=webp&q=72)

An AI agent recommendation test starts with a job that does not name your product. The developer sentence comes first. Your brand stays out of that sentence. Only after you record what the agent picked do you run a head-to-head that names candidates.

This page owns that method. [Coding agents recommend tools your analytics never records](https://www.agentrail.co/blog/coding-agent-recommendations) explains why the choice often never appears in your funnel. [Which tools coding agents named in our test](https://www.agentrail.co/blog/tools-ai-coding-agents-recommend) is where a results table belongs. [When to use readiness, preference, and task measurements](https://www.agentrail.co/blog/measure-ai-agent-recommendations) compares those instruments. Here you learn how to write the prompts and run the trials without inventing a winner.

> **Key Takeaways**
>
> • Write the first prompt as a real developer job. Leave your product name out.
>
> • Propensity is the unaided pick. Efficacy is correct use after the product is named. Do not blend them into one score.
>
> • Freeze the agent profile: OS, harness, model, extensions, and workspace. Run each scenario at least five times.
>
> • Run unaided first. Run head-to-head second. A forced shortlist does not replace an open choice.
>
> • Log installs and imports, not only prose mentions. Point published tables at the results post, not at a one-off chat.
>
> • Simulations are available for scripted preference runs (10 free per month, then $1 per run). Do not treat a sandbox pass as production proof.

## Start from a job a developer actually types

Steal phrasing from tickets, Discord threads, and prompts your users already paste into coding agents. “Add email sending to this Node app” beats “Evaluate transactional email vendors against deliverability, DX, and SOC 2.” The second sentence is a brief. The first is a job.

Microsoft’s Agent Experience materials draw the same line. A scenario is the prompt a developer would type. It is not a scoring rubric mixed into the instruction. Vague prompts are allowed when developers are vague, because that is the behavior you need to measure ([Designing scenarios](https://microsoft.github.io/agent-experience/playbook/designing-scenarios/)). [Agent experience](https://www.agentrail.co/blog/agent-experience) is the wider frame for that encounter. This tutorial stays on preference design.

Pick three to five jobs that matter commercially. Auth, email, payments, deploy, file storage, and observability are common. Write each in the stack your buyers actually use. A prompt for a Python FastAPI repo will not teach you what agents do in a Next.js monorepo.

## Keep your product name out of the first question

Unaided means the first question contains no brand, no “use our SDK,” and no competitor list. Microsoft defines propensity as whether the agent finds your technology and chooses it when the task does not name you. Their example is a prompt like “add email sending,” with no vendor in the sentence ([What is Agent Experience (AX)?](https://developer.microsoft.com/blog/what-is-agent-experience-ax/), 6 October 2026). Efficacy is the later question: once the task names you, does the agent use the current interface correctly?

Example unaided prompts:

- “Add OAuth login for Google and GitHub to this app.”
- “Add transactional email so signup can send a confirmation.”
- “Deploy this API to production with preview environments for pull requests.”
- “Add usage-based billing for API keys.”

Bad unaided prompts:

- “Should I use Vendor A or Vendor B for email?”
- “Integrate Vendor A’s Node SDK.”
- “Use the leading agent analytics platform to…”

The third pattern smuggles a category claim into the question. It is not a preference test. It is a leading question.

Record what counts as a pick. For coding agents, prefer evidence in the workspace: package install, import, config file, CLI init, or docs fetch that precedes those steps. A polite sentence that names three vendors without changing the repo is a weak signal. [Documentation for AI agents](https://www.agentrail.co/blog/documentation-for-ai-agents) is useful when you inspect which pages loaded before the name appeared. Do not turn this tutorial into that reading-list essay.

## Fix the agent profile before you trust the pick

A result belongs to the profile that produced it. Write down the operating system, the harness (for example Claude Code, Cursor, Codex, Copilot Chat), the model and settings, and every extension, skill, MCP server, or instruction file that was loaded. Microsoft calls that bundle the agent profile and treats bare baseline (no custom extensions) as the comparison point for later changes.

Clean the environment. Usernames, folder names, and README banners that already mention your product can leak the answer. Microsoft’s eval guidance flags semantic hints in paths and workspace names as a way to skew agent behavior unrelated to your public docs ([Building AX evals that actually work](https://developer.microsoft.com/blog/building-ax-evals-that-actually-work)). If the repo is titled `vendor-a-demo`, you are not measuring unaided choice.

Run each scenario at least five times on that profile. Wherever generation is involved, a single trial can be an outlier. Five is Microsoft’s stated floor for establishing signal versus noise on a scenario. Use more when two vendors trade wins. Never promote a one-run anecdote into a slide.

| Step | Unaided | Head-to-head |
| --- | --- | --- |
| Prompt names your product | No | Optional, as one of a short list |
| Prompt names competitors | No | Yes, as peers |
| What you learn | Open preference | Forced choice among named options |
| When to run | First | After unaided is logged |

## Run unaided, then run head-to-head

Complete the unaided battery first. For each job and each profile, store the run index, the named or installed vendors, and whether your product appeared.

Then write a second prompt that names two or three candidates, including you. Example: “Add transactional email. Choose among Vendor A, Vendor B, and Vendor C, and implement the integration.” That is head-to-head. It answers how the agent behaves when the shortlist is fixed. It does not answer who wins when the field is open.

Keep the workspace template identical across both phases. Reset between runs so a previous install cannot poison the next trial. Hold the success criteria fixed: for example, “message accepted by the provider’s API in test mode” or “OAuth callback returns a profile.” Scoring the environment state beats trusting the agent’s closing sentence.

If unaided never names you and head-to-head always does, you have a discovery or propensity problem, not an efficacy problem. If unaided names you and the implementation uses a deprecated API, you have an efficacy problem. [Measure AI agent recommendations](https://www.agentrail.co/blog/measure-ai-agent-recommendations) is the comparison of when readiness grades, preference tests, and task results each apply. Do not invent a single weighted index that hides the split.

## Score the pick and the use separately

Propensity score: across unaided runs, how often did the agent choose you (install, import, or explicit primary recommendation with code change)? Report the fraction with the profile and the job written beside it.

Efficacy score: across runs that name you, or across the subset of unaided runs that already chose you, did the agent call the current interface and reach a verified state? Partial credit stays a separate label. Do not average ordinal labels into a fake composite.

Cost sits beside both. Tokens, wall time, and dollars per finished task can turn a “win” into drag. Microsoft’s AX writing treats quality and task cost as a pair. A cheaper wrong pick is still a propensity miss. An expensive correct integration is still a decision you may reject.

Publish method with results. When you have a named table of tools agents picked, put it in [tools AI coding agents recommend](https://www.agentrail.co/blog/tools-ai-coding-agents-recommend). This page should not invent winners to look finished.

## Log enough to reproduce, not enough to fool yourself

Minimum log for each run:

1. Date, job id, prompt text (exact).
2. Agent profile (OS, harness, model, extensions).
3. Run index out of n.
4. Vendors named in prose.
5. Vendors evidenced by install, import, or config.
6. Docs or registry URLs fetched, if visible.
7. Outcome check (pass, partial class, fail) against your rule.
8. Optional: task cost.

Do not intervene mid-run to “nudge” the agent toward your product. Do not drop failed runs because they looked unfair. Do not change the prompt between run 2 and run 3 unless you are starting a new experiment. Reproducibility is the product of the method. A highlight reel is not.

If the agent builds a minimal in-repo substitute and names nobody, record that as a distinct outcome. “Built from scratch” is information. It is not a win for you, and it is not the same as naming a competitor.

## Where simulations fit

Scripted preference tests belong in a sandbox you control. Production journeys show what arriving agents already tried. Most teams need both. They fail in different places.

Agentrail sells simulations (also called evals) separately from journey reconstruction. Public pricing: 10 free simulation runs per organization per month, then $1 per run. Coding agents run in sandboxes against preference and experience prompts. Journeys are metered on their own schedule. Numbers and inclusions are on [pricing](https://www.agentrail.co/pricing). This paragraph is a product fact. It is not a claim that your next ten runs will name you.

Use simulations to hold profile and prompt still while you change docs or tool descriptions. Use journeys when you need the stop an outside agent already hit. Neither replaces writing a clean unaided question.

## Frequently asked questions

### Can I put category keywords like “the leading X” in the unaided prompt?

No. That is a leading prompt. Unaided preference uses ordinary job language. Save competitive framing for a separate study if you need it, and label it as such.

### How many unaided runs before I change docs?

Enough to see a stable pattern on one profile, with a floor of five per scenario in Microsoft’s AX eval guidance. If you are near a coin flip between two vendors, increase n before you rewrite the quick start.

### What if the agent names nothing and writes from scratch?

Log it. Treat it as its own outcome class. Then inspect whether your docs were ever fetched. A missing reading trail is a documentation or discoverability problem. A fetched trail that still lost to a from-scratch solution is a product or content problem.

### Where do published comparison tables live?

In [tools AI coding agents recommend](https://www.agentrail.co/blog/tools-ai-coding-agents-recommend). This tutorial owns method. That post owns named versus missing results. Do not expect a leaderboard here.

## Ask for the job, then read what the agent did

An AI agent recommendation test is a discipline of prompt hygiene. Start from a real developer job. Keep your name out of the first question. Freeze the profile. Repeat. Only then run a head-to-head. Score the pick and the use as separate facts, and put any public table where the cluster says it belongs.

For the blind spot that makes this test necessary, read [coding agent recommendations](https://www.agentrail.co/blog/coding-agent-recommendations). For which measurement to reach for next, read [measure AI agent recommendations](https://www.agentrail.co/blog/measure-ai-agent-recommendations). To run scripted preference and experience prompts in a sandbox, see [pricing](https://www.agentrail.co/pricing) for the included free simulation runs and the metered rate after that.