Why AgentReady Signs Its Crawler With Web Bot Auth

Your store is being read by software you'll never meet. When someone asks ChatGPT for "a linen duvet that ships to Canada" or tells an assistant to "find me running shoes under a hundred dollars," a bot goes and reads product data, including, increasingly, yours. The question every site now has to answer is a simple one with no easy answer: when a request shows up claiming to be a helpful AI agent, how do you know it actually is?

For twenty years the answer was a User-Agent string. A bot announces itself in a header, and you take it at its word. The problem is that the header is a sticky note. Anyone can write "GoogleBot" on it. Scrapers do it constantly, which is why the moment storefronts started getting hammered by AI traffic, the blunt instrument came out: block aggressively, and a lot of legitimate agents got caught in the net alongside the bad ones.

That's the backdrop for a standard worth knowing about, and for a change we just shipped.

A name tag versus a signature

Web Bot Auth is an emerging IETF standard, championed by Cloudflare, that replaces the name tag with a signature. The idea is the kind of obvious that's only obvious in hindsight: instead of claiming to be a particular bot, you prove it.

It works the way secure connections have worked for years. The bot holds a private cryptographic key and publishes the matching public key at a fixed, public address. Every request it makes is signed with the private key. Any site that receives the request fetches the public key, checks the signature, and gets a yes-or-no answer to "did the bot that owns this key really send this?" A scraper can copy the name. It cannot copy the key, so it cannot forge the signature.

Under the hood it's built on RFC 9421, the published HTTP Message Signatures standard, plus a couple of active drafts that define the bot-specific bits. The signature rides in three headers, the key is an Ed25519 keypair, and the public directory is a plain JSON document. Nothing exotic, which is the point: it's the boring, verifiable plumbing the agentic web has been missing.

What we changed

AgentReady reads storefronts. To tell you how ready your store is for AI shopping, we make a handful of read-only fetches: your robots.txt and sitemap, a synthetic crawl of a page or two, your navigation structure. We've always identified ourselves honestly in those requests. As of this week, we don't just say it's us. We prove it.

Every outbound request AgentReady makes to the open web is now signed with Web Bot Auth. Our public key lives at a standard location you can read right now:

https://app.caffeinecommerce.com/.well-known/http-message-signatures-directory

Each signed request carries a Signature-Agent header pointing back at that directory, a Signature-Input describing what was signed, and the Signature itself. If a request claims to be AgentReady, you, or your CDN, can verify it in a few lines of code. If the signature doesn't check out, it isn't us, and you should treat it accordingly.

We did this before anyone asked us to, and that's deliberate. The agentic web only works if the agents in it can be held accountable. A bot that hides behind a forgeable header is asking for trust it hasn't earned. A bot that signs its requests is handing you the receipts.

Why a merchant should care

You don't have to do anything for this to benefit you, which is the best kind of feature. But it's worth understanding what it buys.

As the share of automated traffic climbs, the web is sorting bots into two piles: verifiable and not. Verified-bot programs let sites wave the good ones through and throttle the rest. A crawler that can prove its identity stays in the first pile. One that can't drifts toward the second, the pile that gets rate-limited, challenged, or blocked outright, and a blocked crawler is a store that never gets read. By signing our requests, AgentReady stays legible to your infrastructure instead of becoming one more anonymous bot you have to decide whether to trust.

It also says something about how we think the whole thing should work. We've written a lot about making your store readable to agents, from your robots.txt and AI crawler rules to your agents.md file to the Global Catalog that AI assistants now shop from. Verifiability is the other side of that coin. We ask sites to be open to good agents; the least we can do is be a good agent that's easy to verify.

The short version

A User-Agent string is a claim. A Web Bot Auth signature is proof. AgentReady now signs every request it makes to the open web, our public key is published and verifiable today, and we did it because the agentic web should run on accountable software, not anonymous traffic.

If you want to see where your own store stands with the AI agents reading it, the free AI-readiness checker scores you in a couple of minutes, no install required.