Web Bot Auth:
visitors with no eyes

Crawlers and agents are not the same thing

Here we need a distinction almost everyone skips.

A crawler indexes: it passes through, copies, leaves. That is the thing IT has known for twenty years and blocks on reflex. An agent, instead, acts on behalf of a person right now: it looks for a supplier for whoever just asked a question, checks availability, fills in a quote request. The numbers are misleading: for every real visitor an assistant sends you, its crawler has been past hundreds of thousands of times.

You have to look not (only) at search volume, but at intent. And once you read the intent, you can identify the qualified prospect you can actually profile.

The interesting part is the conversion

Independent measurements from these past months converge on a point that is not obvious: traffic arriving from AI assistants converts 4 to over 20 times better than traditional organic.

There is nothing magical about it. Whoever arrives through an assistant has already given instructions, has probably already compared several alternatives, already knows what to "expect". So the agent turns up briefed, and it does not linger: it acts.

Potentially, it is a channel with real economics, and it gets ignored (almost always) purely because it does not appear on a chart.

Block or open is the wrong question

When this reaches the table, the discussion polarizes within half an hour. On one side, the people who want to shut everything down, because scraping and abuse are expensive. On the other, the people who want to throw the doors open, because the channel is growing.

They are both wrong, so nobody is being singled out here, and for the same reason: they are treating an identity problem as if it were a traffic and resources problem. Blocking agents (through robots.txt and friends) means closing a conversion channel; opening up to them means risking abuse. The question is not who comes in, but who comes in with what "intent".

There is a third path, though, still experimental. It is called Web Bot Auth.

The agent cryptographically signs its own request with HTTP Message Signatures (RFC 9421), an Ed25519 key and a header pointing to the public key directory. You verify the signature and know who you are talking to before deciding what to grant. This is not a niche experiment: Cloudflare, Amazon, Akamai and OpenAI are behind it, an IETF working group was chartered this year, and the payment networks are adopting it as their front door.

The real question is not who you are, it is who sent you

This is where it gets interesting, because verifying the agent solves only half the problem.

An agent that books, buys or opens a ticket is not acting for itself. It is acting on a delegation. The questions your application has to answer are at least four, not one: who are you, who authorized you, up to what amount (or under which other restrictions), until when.

The direction already exists and is being built. AP2, the agentic payments protocol that emerged this year under the FIDO Alliance, breaks authorization into three signed mandates: the user's intent, the cart approved at that price, the payment. Each one is a verifiable credential with explicit, clearly drawn limits.

If it sounds familiar, that is because it is: it is Zero Trust applied to an actor that is not a person. The difference is that this time the actor moves at machine speed and never gets tired.

So, does it all have to be rebuilt?

No, the product does not need rewriting. What needs to go is the idea that the graphical interface is the only way to use your software. By now it is obvious that it is not.

1. Measure server-side, not in the browser. Application logs see what the script cannot. Separate crawlers from active agents, and take that number into the same slide where you take sessions, views and the rest. Until it has a number, it will not have a budget.

2. Expose capabilities as contracts, not as pages. Catalog, availability, pricing, booking: if an agent has to infer them by reading the DOM, your next restyling cuts you out. Contract-first APIs, structured data and, where it makes sense, an MCP endpoint (aka WebMCP).

3. Separate identity from delegation. Verify the agent with its signature, and treat the user's authorization as a distinct thing, with scope, spending cap and expiry. An authenticated agent is not (necessarily) an authorized one.

4. Classify and recognize. A badly defined block or rate limit carries the weight of a commercial decision, so it deserves proper thought.

A side effect: the stress test

There is one thing we have noticed: when an agent cannot complete an action on your product, almost always that action is hard work for a person too. A booking flow with four ambiguous steps is opaque to everyone. Agentic traffic is not just a channel, it is a validation you should not dismiss.

And here are the traps not to fall into:

- A frontend-only shop will chase a prettier site that agents will struggle to navigate.

- A security-only shop will try to isolate and block anything that is not human.

- An AI-only shop hands you a chatbot, because that way you are more "smart".

The truth is simply that you have to start from the product's capabilities, so you can make it usable, verifiable and delegable.. again, for everyone.