What is Web Bot Auth?
A way for a bot to prove who it is. The bot's operator publishes its public keys on its own domain and signs every request it sends; a site that receives one can check the signature against those keys. It is being standardised at the IETF, and it isn't finished yet.
The problem it solves
A bot has always announced itself with a user agent name, and anyone can type any name. Sites, and the firewalls in front of them, have filled the gap with lists of the IP addresses each company says its crawlers use — lists that break whenever a company moves, and say nothing about a bot running on shared cloud addresses.
A signature can't be copied the way a name can. Only the holder of the private key can make one that verifies, so a request that checks out really did come from the operator it names.
How it works
It is built on HTTP Message Signatures, a published standard (RFC 9421, February 2024) for signing parts of an HTTP request. Web Bot Auth adds the pieces a bot needs on top of it.
The operator keeps a key directory: a file of its public keys in the JSON Web Key Set format, at a well-known address on its own domain, /.well-known/http-message-signatures-directory. Each request then carries a signature over chosen parts of the request, with its own creation and expiry time, and a Signature-Agent header saying which directory to check it against. A site reads the header, fetches the directory, and verifies.
Where it stands
It began as an individual Internet-Draft and is now the work of an IETF working group, also called Web Bot Auth. The group's protocol draft was last updated in September 2026. A draft is a work in progress, not a standard, and the details can still change.
It is already in use. Cloudflare checks these signatures to recognise verified bots and signed agents, so a site behind Cloudflare can let a signed bot through without anyone keeping an IP list by hand.
Who needs one
Only a domain that runs a bot or an agent. An ordinary website has nothing to publish: it is on the receiving end, where the firewall or CDN in front of it may already check signatures for it. That's why our Advanced check for it expects to find nothing on most sites, and says so.
Being honest about it
A valid signature proves who sent a request, and nothing more. It doesn't make a bot polite, and it doesn't tell you what the bot will do with your page. What it ends is the guessing: a site can decide about a bot it knows, rather than one that merely says its name.
Checks that measure it
Further reading
Related terms
Not sure where yours stands?
A free scan checks your page the way an answer engine reads it, in about 20 seconds — no signup.
Rather have it handled? We do that too. No pitch, just a plain-English chat.
Book a call