---
title: "What agent behavior looks like in production"
description: "Agent behavior is the path a task takes across docs, APIs, SDKs, and MCP. Most AI bot traffic is still a crawl, not a task a person sent."
canonical: "https://www.agentrail.co/blog/agent-behavior"
author: "Vivek Mittal"
published: "2026-10-17T00:00:00Z"
last_updated: "2026-10-23T10:23:08.159Z"
---

# What agent behavior looks like in production

Agent behavior is the path a task takes across docs, APIs, SDKs, and MCP. Most AI bot traffic is still a crawl, not a task a person sent.

![Agent Behavior title with footprint icons on a green gradient](https://images.ctfassets.net/1htfyfnn1o5y/5pkXF29rf6UYzhPyl5hr5/1182f189da6b1c24c317db2ca836c9a0/hero_image.png?w=1600&fm=webp&q=72)

A person arrives on a homepage, reads a headline, and maybe books a demo. An agent arrives with a job. It calls a tool, reads the result, and chooses the next call from that result.

That path is agent behavior. This piece is about the paths you can see once the agent is on your product, not about how to build the agent.

> **Key Takeaways**
>
> Agent behavior is the path one task takes. It is not a pageview count and not a bot-traffic chart.
>
> In production, the agent picks the next step from the last tool result, and it stops when the task ends or a limit trips.
>
> Cloudflare’s 2025 crawl data puts user-action crawling at about 2–3% of AI crawling. Training still dominates. A crawler chart is not a task chart.
>
> The paths people skip are normal here: a markdown index, an OpenAPI file, an `AGENTS.md`, then an API or MCP call, often with retries.
>
> Your session log will split that single task across docs, the API, the SDK, and MCP unless you put it back together.

## Agent behavior is the path a task takes

Agent behavior is the sequence of steps an agent takes while it tries to finish a task on your product.

The steps are concrete. Fetch a docs page. Read an error body. Retry with a different argument. Call an MCP tool. Create a key. The behavior is that sequence, in order, until the task ends.

A pageview total cannot say this. Ten thousand fetches might be one task repeated, or a crawler that never had a goal. The unit that makes the path readable is the task. How those requests get grouped is a later problem. [Agent journey analytics](https://www.agentrail.co/blog/agent-journey-analytics) is where that stitch is defined.

People still matter. Someone typed the prompt, or a job started the agent on a schedule. The behavior you can observe is still the agent’s path, not the person’s cursor.

## Production agents pick the next step from the last tool result

[Anthropic’s December 2024 engineering note](https://www.anthropic.com/engineering/building-effective-agents) draws a useful line. Workflows are code paths you wrote in advance. Agents are systems where the model directs its own process and its own tool use.

In that loop the agent needs ground truth from the environment on each turn. A tool result, a test failure, or an HTTP status is the ground truth. The model’s plan is not. If the status is 401, the next step is often auth, not the happy-path call the prompt imagined.

The same note says a task often ends when the work is done, and that teams also set a stop, such as a maximum number of iterations. So production behavior includes quitting. An agent that hammers one endpoint fifty times is showing a path, and the path includes a missing stop or a missing fix.

This is not a claim about every framework. It is the shape Anthropic reported from its own agents and from customer builds: a model, tools, and a check against the world after each call.

## Most recorded AI traffic is a crawl, not a task

If you open a bot report and treat every AI request as agent behavior, you will misread your product.

[Cloudflare’s 29 August 2025 analysis](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) of AI crawling split purpose from operator disclosures. Over the prior twelve months, 80% of that crawling was for training, 18% was for search, and 2% was for user actions. In the last six months of the same window, training was 82%, search was 15%, and user actions were 3%.

Year over year, inside that post, training’s share moved from 72% in July 2024 to 79% in July 2025. User actions moved from 2% to 3.2%. Search fell from 26% to 17%.

| Purpose of AI crawling | Prior 12 months | Last 6 months of that window |
| --- | --- | --- |
| Training | 80% | 82% |
| Search | 18% | 15% |
| User actions | 2% | 3% |

Source: Cloudflare, 29 August 2025. Purpose is classified from operator disclosures, not from task success on your product.

Volume moved as well. April 2025 crawling was up 32% versus May 2024. By July 2025, year-over-year growth was 4%. Inside AI and search crawler traffic, GPTBot’s share went from 4.7% to 11.7% between July 2024 and July 2025. ClaudeBot went from 6% to 9.9%. ChatGPT-User, closer to a person-triggered fetch, went from 0.1% to 0.9%.

Those figures are about crawls Cloudflare can name. They are not a completion rate, and they are not the share of your signups that agents drove. They do one job: they stop you from reading a training crawl as a buyer finishing a task.

User-action traffic is the slice that can look like behavior. It is still small in that dataset, and it is still a crawl class. A coding agent on your docs may never wear a public crawler name at all.

## Agents fetch a map, then a page, then a contract

Once a task is real, the path is often a chain of files a person would not open first.

[Jeremy Howard’s `llms.txt` proposal](https://llmstxt.org/), first published 3 September 2024 and updated 10 August 2026, says agents use websites constantly and that HTML is a poor packet for them. Navigation, ads, and JavaScript sit around the text. Context is finite, so a curated markdown file at a known path is easier than reconstructing the page. The proposal expects the agent to read `llms.txt`, then follow links to markdown. It also says a sitemap is the wrong substitute, because a sitemap lists too much and usually lists the human URLs.

An API path looks similar, one step later. [The OpenAPI Specification](https://spec.openapis.org/oas/latest.html) defines a language-agnostic description of an HTTP API so a computer can discover the service without source code and without watching traffic. Spec 3.2.1 is dated 10 September 2026. An agent that finds that file will call from the contract. An agent that cannot find it will guess paths from prose, or from whatever the docs page rendered.

Coding agents add a file of their own. [The `AGENTS.md` site](https://agents.md/), retrieved 17 October 2026, says the format is used by over 60,000 open-source projects. It is a README aimed at the agent: build steps, tests, conventions. The site also says that if you list test commands, the agent will try to run them. That is behavior. You published a command, and the task grew a shell step.

MCP sits at the end of many of these paths. Whether the agent can find the server is [agent discoverability](https://www.agentrail.co/blog/agent-discoverability). The behavior, once it has found the server, is a list call and then the tool calls the task actually needs. Failed tools show up as repeats, not as a bounce in a funnel report.

## The paths a person does not take

Human analytics assumes a funnel. Land, consider, sign up, activate. Agents do not owe you that order.

A single task can start in documentation, inspect an SDK, call an MCP tool, hit an API, and only then authenticate. Agentrail’s product description is built around that pattern, because the product surfaces record it as separate hits. The agent experienced one job.

Anthropic also describes parallelization: independent subtasks run at the same time, or the same task run more than once so the results can be compared. On your side that looks like overlapping requests, not a neat session of one page after another.

Retries are part of the path too. A 404, a schema mismatch, or an empty body sends the agent back. People abandon. Agents often try a nearby URL or a second argument shape, until the iteration cap hits.

None of this requires a dramatic traffic share. A handful of tasks is enough to see a path your funnel was not drawn for. Early journeys are useful for that reason. They show the assumption that broke, before you have a crowd.

## The interface you publish is the behavior you get

Agents use the interface in front of them. If the interface is awkward, the path gets awkward. That is not a slogan. It is what showed up when Anthropic tuned tools for a coding agent.

In that December 2024 note, the team says the model made mistakes with tools that took relative file paths after the agent had left the root. They changed the tool so it required absolute paths. The model then used that form without the same mistakes. They also say they spent more time on the tools than on the overall prompt.

Do not generalize that anecdote into a rule that every relative path fails. Do take the mechanism. The agent will do what the tool definition, the error body, and the docs make easy. Vague parameter names, two tools that overlap, and errors that say only “bad request” all become visible behavior: extra calls, wrong arguments, and a stop.

The same loop explains quiet success. A small tool surface, a stable error code, and a docs page that matches the code let the agent finish in fewer steps. You see a short path. That short path is the behavior worth keeping.

## A session log splits one task into several properties

Standard product analytics will store this activity. It will not automatically store it as one behavior.

The docs host has a pageview. The API gateway has a client id, or none. The MCP process has a tool name. The SDK example might only show up as a clone, or as nothing, if the agent never executed in your tenant. Each system is telling the truth about its own events. None of them was asked which events were the same task.

So the practical watchlist is short. Which URL or tool came first. Which error repeated. Where the path stopped. Whether auth happened before or after the failing call. Whether the agent fetched markdown or only the HTML shell.

That is enough to see behavior. It is not the whole measurement set. Rates, cost, and the rest of the stack belong in [the measurement guide](https://www.agentrail.co/blog/how-to-measure-agent-experience), not here. The frame for the experience itself is [agent experience](https://www.agentrail.co/blog/agent-experience): what the agent was trying to do, and where that attempt failed.

We built Agentrail to reconstruct that path across website, docs, APIs, SDKs, and MCPs. The reconstruction is the point of this article’s question. A crawler percentage cannot answer it.

## FAQs

### What is agent behavior?

Agent behavior is the sequence of steps an agent takes to finish a task on a product. The sequence includes fetches, tool calls, errors, retries, and the stop. It is not a count of bot requests.

### Are AI crawlers and task agents the same traffic?

No. Cloudflare’s August 2025 breakdown put training at about 80% of AI crawling and user actions at about 2–3%. Even the user-action class is a crawl purpose, not proof that a task completed on your product.

### Why do agents ignore the homepage?

Often the task does not need it. `llms.txt`, an OpenAPI file, or `AGENTS.md` is a shorter route to the fact the agent needs. The homepage is written for a person comparing vendors.

### Does a long path mean the agent is low quality?

Not by itself. A long path can mean a hard task, a missing document, or a tool that invites mistakes. Read the repeated step before you blame the model. Anthropic’s path fix was a tool change, not a model swap.

## Conclusion

Production agent behavior is a task-shaped path: tool call, result, next call, stop. Most AI request volume you will see in a public crawl report is still training, so separate that chart from the tasks people actually send.

Watch the files agents open and the errors they repeat. Then put the split hits back into one path. [Agent experience](https://www.agentrail.co/blog/agent-experience) is the name for that reading of your product. If you want the path rather than another session, start there.