Bot Appétit State of the Web & AI Readiness
How many of the sites we scan let our scanner in, and what they get right and wrong, check by check. Percentages only, refreshed every hour.
Updated 20:07 UTC · October 2026
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Headlines
Headlines show up once a result has held steady for two days, so one odd hour never makes the news.
Can AI read the web?
In October 2026, our scanner could read 90% of the sites we scanned. The other 10% turned it away.
This is our scanner, not every AI. A site that turns it away may still let other visitors in; how sites treat AI crawlers is in AI access, below.
How sites land
| Verdict | Share of sites |
|---|---|
| Chef's kiss | 20% |
| Cleans the plate | 43% |
| Needs seasoning | 18% |
| Undercooked | 8% |
| Sent back to the kitchen | 0% |
| Turned our scanner away | 10% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Embed or share this figure
AI access
In October 2026, 8% of the sites we scanned disallowed some AI crawlers in robots.txt, and 10% turned our scanner away at the server.
| What we found | Share of sites |
|---|---|
| robots.txt disallows some AI crawlers | 8% |
| robots.txt disallows major AI crawlers | 5% |
| The server turned our scanner away | 10% |
| robots.txt allows AI crawlers, the server turned our scanner away | 4% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Embed or share this figure
Embed or share this figure
Where sites differ
In October 2026, here is where the sites we scanned differ most.
By platform
In October 2026, the “OpenID Connect discovery” check showed the widest gap between platforms: 88% of Shopify sites passed it, against 0% of Webflow sites.
| Platform | Disallow some AI crawlers in robots.txt |
|---|---|
| Next.js | 13% |
| Shopify | 11% |
| Webflow | 1% |
| WordPress | 8% |
A platform is only known for pages our scanner could read, so this table cannot say how many sites on each platform turned our scanner away.
The checks that differ most between platforms, by the share of sites that pass.
| Check | Next.js | Shopify | Webflow | WordPress |
|---|---|---|---|---|
| OpenID Connect discovery | 5% | 88% | 0% | 3% |
| OAuth protected-resource metadata | 5% | 88% | 1% | 10% |
| OAuth authorization-server metadata | 7% | 88% | 2% | 10% |
| Open Graph tags for sharing | 60% | 85% | 2% | 62% |
| Images are optimised | 44% | 79% | 1% | 33% |
By hosting and CDN
In October 2026, 14% of the sites we scanned behind Amazon CloudFront turned our scanner away at the server, and 0% allowed AI crawlers in robots.txt but turned our scanner away anyway.
| CDN | Turned our scanner away at the server | Allow AI crawlers in robots.txt and still turned our scanner away |
|---|---|---|
| Amazon CloudFront | 14% | 0% |
| Cloudflare | 10% | 4% |
| Fastly | 7% | 2% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Every check
In October 2026, “Not flagged for malware or phishing” was the check sites passed most often, at 100%. “llms.txt is present and valid” was the one they failed most often, at 64%.
A dash means not enough sites yet. Each check links to what it is and what good looks like.
Fixed and Broke count re-scanned sites: Fixed went from not passing to passing, Broke went the other way. Not enough history yet.
AEO / content
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Your content survives without JavaScript | 93% | 1% | 3% | 3% | — | — |
| Your answer is in the HTML | 92% | 3% | 4% | 0% | — | — |
| Semantic HTML landmarks | 58% | 42% | 0% | <1% | — | — |
| Content is scannable in self-contained chunks | 89% | 11% | 0% | <1% | — | — |
| Author / publisher is clear | 57% | 43% | 0% | <1% | — | — |
| A machine-readable published / updated date | 26% | 74% | 0% | <1% | — | — |
| llms.txt is present and valid | 31% | 1% | 64% | 5% | — | — |
| Clean content for agents (Markdown) | 7% | 83% | 0% | 10% | — | — |
AI access
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Buttons, links and fields have names | 53% | 44% | 3% | 0% | — | — |
| AI crawlers aren't blocked | 85% | 8% | 4% | 3% | — | — |
| Automated visitors can reach your pages | 100% | 0% | 0% | 0% | — | — |
| Engines are allowed to quote the page | 98% | 2% | <1% | 0% | — | — |
| Your page arrives quickly | 89% | 7% | 4% | 0% | — | — |
AI signals
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| AI bot rules are explicit and intentional | 20% | 77% | 0% | 3% | — | — |
Structured data
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Structured data is present | 64% | 0% | 35% | <1% | — | — |
| Structured data is valid | 98% | 1% | 1% | <1% | — | — |
| FAQ / Q&A schema | 52% | 48% | 0% | 0% | — | — |
| Organization schema with sameAs | 31% | 22% | 47% | <1% | — | — |
| Right schema type for the page | 89% | 11% | 0% | <1% | — | — |
| Schema facts also appear in the page text | 57% | 43% | 0% | 0% | — | — |
| Breadcrumb structured data | 30% | 70% | 0% | 0% | — | — |
SEO
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Reached without a redirect chain | 98% | 2% | 0% | 0% | — | — |
| Caching or validation headers are set | 88% | 12% | 0% | 0% | — | — |
| The site settles on one host (www vs non-www) | 75% | 8% | 7% | 11% | — | — |
| robots.txt is present and not over-blocking | 88% | <1% | 11% | 1% | — | — |
| Not accidentally set to noindex | 98% | 0% | 2% | 0% | — | — |
| This page declares its canonical URL | 72% | 6% | 22% | 0% | — | — |
| Has a title tag | >99% | <1% | <1% | 0% | — | — |
| Has a meta description | 73% | 14% | 13% | 0% | — | — |
| The page has an H1 | 71% | 4% | 24% | <1% | — | — |
| Logical heading order | 61% | 31% | 4% | 4% | — | — |
| Topic is consistent across title, H1 and description | 55% | 14% | 0% | 31% | — | — |
| Images have descriptive alt text | 87% | 10% | 3% | <1% | — | — |
| Your page's HTML isn't oversized | 98% | 1% | 1% | 0% | — | — |
| Open Graph tags for sharing | 53% | 32% | 15% | 0% | — | — |
| XML sitemap is present and referenced | 72% | 8% | 13% | 6% | — | — |
| Missing pages return a real 404 | 85% | 0% | 9% | 6% | — | — |
| Language and charset declared | 93% | 7% | 0% | 0% | — | — |
| hreflang for language variants | 98% | 2% | 0% | 0% | — | — |
| No meta-refresh redirect | >99% | <1% | 0% | 0% | — | — |
Links
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| No broken links | 84% | 8% | <1% | 8% | — | — |
| Links don't point at redirects | 83% | 9% | 0% | 8% | — | — |
| Descriptive anchor text | 99% | 1% | 0% | <1% | — | — |
| Links are real links | 89% | 10% | 1% | 0% | — | — |
Performance
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Largest Contentful Paint (LCP) | — | — | — | — | — | — |
| Cumulative Layout Shift (CLS) | — | — | — | — | — | — |
| Interaction to Next Paint (INP) | — | — | — | — | — | — |
| Page weight and render-blocking | — | — | — | — | — | — |
Security
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Served over a secure HTTPS connection | 99% | 0% | 1% | 0% | — | — |
| HTTP redirects to HTTPS | 87% | 3% | 7% | 3% | — | — |
| No mixed (insecure) content | 98% | 1% | <1% | <1% | — | — |
| Not flagged for malware or phishing | 100% | 0% | 0% | 0% | — | — |
Advanced
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Content usage preferences declared | 7% | 90% | 0% | 3% | — | — |
| Resource-discovery Link headers | 4% | 96% | 0% | 0% | — | — |
| MCP Server Card | 3% | 84% | 0% | 13% | — | — |
| AI catalog | 2% | 88% | 0% | 10% | — | — |
| A2A Agent Card | 1% | 89% | 0% | 9% | — | — |
| WebMCP tools in the page | 1% | 99% | 0% | <1% | — | — |
| AI usage preferences (Content-Usage) | <1% | 98% | 0% | 2% | — | — |
| Licensing terms for AI (RSL) | <1% | 98% | 0% | 2% | — | — |
| Text-and-data-mining reservation (TDMRep) | <1% | 91% | 0% | 9% | — | — |
| Markdown version linked from the page | 4% | 96% | 0% | <1% | — | — |
| Content signals agree everywhere | — | — | — | — | — | — |
| Agent Skills index | 2% | 87% | 0% | 11% | — | — |
| API catalog | 2% | 86% | 0% | 12% | — | — |
| OAuth authorization-server metadata | 8% | 79% | 0% | 13% | — | — |
| OAuth protected-resource metadata | 6% | 80% | 0% | 14% | — | — |
| OpenID Connect discovery | 5% | 81% | 0% | 14% | — | — |
| Web Bot Auth (operates a signed bot) | <1% | 85% | 0% | 14% | — | — |
| Universal Commerce Protocol manifest | — | — | — | — | — | — |
| Auth.md agent registration | 1% | 84% | 0% | 15% | — | — |
| AI discovery via DNS | <1% | 99% | 0% | 1% | — | — |
Good practice
| Check | Pass | Worth a look | To fix | Couldn't judge | Fixed | Broke |
|---|---|---|---|---|---|---|
| Core security headers are set | 19% | 50% | 30% | 0% | — | — |
| No server version leakage | 84% | 16% | 0% | 0% | — | — |
| security.txt is published | 9% | 83% | 0% | 8% | — | — |
| Mobile viewport is set | 87% | 10% | 3% | 0% | — | — |
| Title fits a search snippet | 54% | 46% | 0% | 0% | — | — |
| Description fits a search snippet | 70% | 30% | 0% | 0% | — | — |
| Twitter/X card tags | 72% | 28% | 0% | 0% | — | — |
| Images are optimised | 31% | 69% | 0% | 0% | — | — |
| Analytics is installed | 60% | 40% | 0% | 0% | — | — |
| Search Console / Bing verification | 74% | 25% | 0% | <1% | — | — |
| A favicon is set | 96% | 4% | 0% | 0% | — | — |
| Claims come with figures, quotes or sources | 43% | 56% | 0% | <1% | — | — |
| Section headings are phrased as questions | 29% | 71% | 0% | <1% | — | — |
| Sensible internal/external link mix | 95% | 5% | 0% | <1% | — | — |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
What goes together
In October 2026, these were the pairs of checks that sites tended to pass together.
This is an association between checks, not a cause: passing one does not make a site pass the other. It shows what tends to sit together on a site that has been looked after.
- Sites that pass the “Author / publisher is clear” check are 121 times as likely to pass the “Organization schema with sameAs” check too.
- Sites that pass the “Clean content for agents (Markdown)” check are 96 times as likely to pass the “A2A Agent Card” check too.
- Sites that pass the “Clean content for agents (Markdown)” check are 82 times as likely to pass the “Markdown version linked from the page” check too.
- Sites that pass the “Clean content for agents (Markdown)” check are 79.5 times as likely to pass the “Agent Skills index” check too.
- Sites that pass the “Resource-discovery Link headers” check are 71 times as likely to pass the “API catalog” check too.
- Sites that pass the “Resource-discovery Link headers” check are 65.5 times as likely to pass the “Markdown version linked from the page” check too.
- Sites that pass the “Resource-discovery Link headers” check are 61 times as likely to pass the “Auth.md agent registration” check too.
- Sites that pass the “Resource-discovery Link headers” check are 55 times as likely to pass the “Agent Skills index” check too.
- Sites that pass the “Clean content for agents (Markdown)” check are 45.5 times as likely to pass the “API catalog” check too.
- Sites that pass the “Content usage preferences declared” check are 44 times as likely to pass the “Agent Skills index” check too.
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
By category
In October 2026, here is how the sites we scanned did in each category we score.
AEO / content
Of the AEO / content results, 57% are passes, 32% are worth a look and 9% need fixing.
AI access
Of the AI access results, 87% are passes, 10% are worth a look and 2% need fixing.
AI signals
Of the AI signals results, 20% are passes, 77% are worth a look and 0% need fixing.
Structured data
Of the Structured data results, 63% are passes, 15% are worth a look and 22% need fixing.
SEO
Of the SEO results, 81% are passes, 8% are worth a look and 7% need fixing.
Links
Of the Links results, 89% are passes, 7% are worth a look and <1% need fixing.
Security
Of the Security results, 96% are passes, 1% are worth a look and 2% need fixing.
| Category | Pass | Worth a look | To fix | Couldn't judge |
|---|---|---|---|---|
| AEO / content | 57% | 32% | 9% | 2% |
| AI access | 87% | 10% | 2% | 1% |
| AI signals | 20% | 77% | 0% | 3% |
| Structured data | 63% | 15% | 22% | <1% |
| SEO | 81% | 8% | 7% | 3% |
| Links | 89% | 7% | <1% | 4% |
| Security | 96% | 1% | 2% | 1% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Structured data
In October 2026, the most common type of structured data was WebSite, on 43% of the sites we scanned.
| Type | Share of sites |
|---|---|
| WebSite | 43% |
| Organization | 38% |
| WebPage | 26% |
| BreadcrumbList | 15% |
| ImageObject | 10% |
| FAQPage | 8% |
| Person | 7% |
| LocalBusiness | 6% |
| SoftwareApplication | 5% |
| Article | 4% |
| Place | 3% |
| ProfessionalService | 3% |
| ItemList | 3% |
| Service | 2% |
| CollectionPage | 2% |
| NewsMediaOrganization | 2% |
| NewsArticle | 1% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Emerging standards
In October 2026, “llms.txt is present and valid” was the emerging standard that sites passed most often: 31% of the sites we scanned passed it.
| Standard | Share of sites |
|---|---|
| llms.txt is present and valid | 31% |
| Clean content for agents (Markdown) | 7% |
| Content usage preferences declared | 7% |
| Resource-discovery Link headers | 4% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Embed or share this figure
Change over time
These lines count the sites scanned in each calendar month, so a month's figure can differ from the headline figures above, which use each site's latest scan from the last three months. The "All sites" column is every site scanned that month. "Same sites" follows only the sites scanned in both months, so a change there is the sites themselves changing, not a different crowd scanning.
AI crawlers aren't blocked
In October 2026, 89% of the sites we scanned passed the “AI crawlers aren't blocked” check, against 78% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 78% | — |
| September 2026 | 81% | — |
| October 2026 | 89% | — |
FAQ / Q&A schema
In October 2026, 57% of the sites we scanned passed the “FAQ / Q&A schema” check, against 30% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 30% | — |
| September 2026 | 43% | — |
| October 2026 | 57% | — |
Structured data is present
In October 2026, 69% of the sites we scanned passed the “Structured data is present” check, against 57% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 57% | — |
| September 2026 | 61% | — |
| October 2026 | 69% | — |
llms.txt is present and valid
In October 2026, 35% of the sites we scanned passed the “llms.txt is present and valid” check, against 30% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 30% | — |
| September 2026 | 28% | — |
| October 2026 | 35% | — |
Organization schema with sameAs
In October 2026, 32% of the sites we scanned passed the “Organization schema with sameAs” check, against 33% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 33% | — |
| September 2026 | 31% | — |
| October 2026 | 32% | — |
XML sitemap is present and referenced
In October 2026, 74% of the sites we scanned passed the “XML sitemap is present and referenced” check, against 69% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 69% | — |
| September 2026 | 71% | — |
| October 2026 | 74% | — |
Breadcrumb structured data
In October 2026, 36% of the sites we scanned passed the “Breadcrumb structured data” check, against 29% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | — | — |
| September 2026 | 29% | — |
| October 2026 | 36% | — |
Organization markup
In October 2026, 42% of the sites we scanned published Organization structured data, against 30% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 30% | — |
| September 2026 | 35% | — |
| October 2026 | 42% | — |
WebSite markup
In October 2026, 47% of the sites we scanned published WebSite structured data, against 33% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 33% | — |
| September 2026 | 38% | — |
| October 2026 | 47% | — |
WebPage markup
In October 2026, 29% of the sites we scanned published WebPage structured data, against 21% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 21% | — |
| September 2026 | 23% | — |
| October 2026 | 29% | — |
SoftwareApplication markup
In October 2026, 5% of the sites we scanned published SoftwareApplication structured data, against 7% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 7% | — |
| September 2026 | 5% | — |
| October 2026 | 5% | — |
Service markup
In October 2026, 2% of the sites we scanned published Service structured data, the same as when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 2% | — |
| September 2026 | 1% | — |
| October 2026 | 2% | — |
FAQPage markup
In October 2026, 11% of the sites we scanned published FAQPage structured data, against 4% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 4% | — |
| September 2026 | 6% | — |
| October 2026 | 11% | — |
Article markup
In October 2026, 4% of the sites we scanned published Article structured data, against 3% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 3% | — |
| September 2026 | 4% | — |
| October 2026 | 4% | — |
TechArticle markup
In October 2026, 1% of the sites we scanned published TechArticle structured data, the same as when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 1% | — |
| September 2026 | 1% | — |
| October 2026 | 1% | — |
NewsMediaOrganization markup
In October 2026, 1% of the sites we scanned published NewsMediaOrganization structured data, against 4% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 4% | — |
| September 2026 | 3% | — |
| October 2026 | 1% | — |
ItemList markup
In October 2026, 3% of the sites we scanned published ItemList structured data, against 2% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 2% | — |
| September 2026 | 3% | — |
| October 2026 | 3% | — |
Person markup
In October 2026, 8% of the sites we scanned published Person structured data, against 7% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 7% | — |
| September 2026 | 7% | — |
| October 2026 | 8% | — |
BreadcrumbList markup
In October 2026, 14% of the sites we scanned published BreadcrumbList structured data, against 7% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 7% | — |
| September 2026 | 15% | — |
| October 2026 | 14% | — |
Place markup
In October 2026, 3% of the sites we scanned published Place structured data, against <1% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | <1% | — |
| September 2026 | 2% | — |
| October 2026 | 3% | — |
ImageObject markup
In October 2026, 11% of the sites we scanned published ImageObject structured data, against 4% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 4% | — |
| September 2026 | 9% | — |
| October 2026 | 11% | — |
LocalBusiness markup
In October 2026, 9% of the sites we scanned published LocalBusiness structured data, against 2% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 2% | — |
| September 2026 | 4% | — |
| October 2026 | 9% | — |
ProfessionalService markup
In October 2026, 5% of the sites we scanned published ProfessionalService structured data, against 3% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 3% | — |
| September 2026 | 2% | — |
| October 2026 | 5% | — |
CollectionPage markup
In October 2026, 2% of the sites we scanned published CollectionPage structured data, against 4% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 4% | — |
| September 2026 | 3% | — |
| October 2026 | 2% | — |
Corporation markup
In October 2026, 1% of the sites we scanned published Corporation structured data, against 2% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 2% | — |
| September 2026 | 1% | — |
| October 2026 | 1% | — |
ProfilePage markup
In October 2026, 1% of the sites we scanned published ProfilePage structured data, the same as when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 1% | — |
| September 2026 | 1% | — |
| October 2026 | 1% | — |
WebApplication markup
In October 2026, 1% of the sites we scanned published WebApplication structured data, against <1% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | <1% | — |
| September 2026 | <1% | — |
| October 2026 | 1% | — |
NewsArticle markup
In October 2026, <1% of the sites we scanned published NewsArticle structured data, the same as when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | <1% | — |
| September 2026 | 2% | — |
| October 2026 | <1% | — |
HomeAndConstructionBusiness markup
In October 2026, 1% of the sites we scanned published HomeAndConstructionBusiness structured data, against 0% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 0% | — |
| September 2026 | <1% | — |
| October 2026 | 1% | — |
EducationalOrganization markup
In October 2026, 1% of the sites we scanned published EducationalOrganization structured data, against 0% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 0% | — |
| September 2026 | <1% | — |
| October 2026 | 1% | — |
SiteNavigationElement markup
In October 2026, <1% of the sites we scanned published SiteNavigationElement structured data, against 0% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 0% | — |
| September 2026 | 1% | — |
| October 2026 | <1% | — |
RealEstateAgent markup
In October 2026, 1% of the sites we scanned published RealEstateAgent structured data, against 0% when this series began.
| Month | All sites | Same sites |
|---|---|---|
| July 2026 | — | — |
| August 2026 | 0% | — |
| September 2026 | <1% | — |
| October 2026 | 1% | — |
Lines that start in October 2026
These lines have one month of figures so far. Each becomes a month-by-month line once a second month of figures arrives.
| Series | All sites |
|---|---|
| Agent Skills index | 2% |
| API catalog | 2% |
| Auth.md agent registration | 2% |
| AI discovery via DNS | <1% |
| MCP Server Card | 3% |
| OAuth authorization-server metadata | 8% |
| OAuth protected-resource metadata | 7% |
| OpenID Connect discovery | 5% |
| Web Bot Auth (operates a signed bot) | <1% |
| AI bot rules are explicit and intentional | 19% |
| Analytics is installed | 55% |
| Descriptive anchor text | 99% |
| Author / publisher is clear | 61% |
| Automated visitors can reach your pages | 100% |
| No broken links | 84% |
| Caching or validation headers are set | 86% |
| This page declares its canonical URL | 72% |
| The site settles on one host (www vs non-www) | 80% |
| Content is scannable in self-contained chunks | 91% |
| Your answer is in the HTML | 93% |
| Content usage preferences declared | 8% |
| Missing pages return a real 404 | 85% |
| A favicon is set | 95% |
| A machine-readable published / updated date | 24% |
| Logical heading order | 62% |
| Reached without a redirect chain | 99% |
| Served over a secure HTTPS connection | >99% |
| HTTP redirects to HTTPS | 86% |
| Images have descriptive alt text | 92% |
| Images are optimised | 33% |
| Topic is consistent across title, H1 and description | 63% |
| Language and charset declared | 93% |
| Resource-discovery Link headers | 5% |
| Sensible internal/external link mix | 95% |
| Clean content for agents (Markdown) | 7% |
| Has a meta description | 88% |
| No meta-refresh redirect | >99% |
| Not accidentally set to noindex | 98% |
| No mixed (insecure) content | 99% |
| Open Graph tags for sharing | 52% |
| Links don't point at redirects | 85% |
| robots.txt is present and not over-blocking | 89% |
| Not flagged for malware or phishing | 100% |
| Search Console / Bing verification | 71% |
| Core security headers are set | 20% |
| security.txt is published | 10% |
| Semantic HTML landmarks | 62% |
| No server version leakage | 83% |
| The page has an H1 | 81% |
| Has a title tag | >99% |
| Twitter/X card tags | 71% |
| Mobile viewport is set | 89% |
| hreflang for language variants | 98% |
| Structured data is valid | 98% |
| Right schema type for the page | 93% |
| Title fits a search snippet | 59% |
| AI catalog | 2% |
| Your content survives without JavaScript | 93% |
| A2A Agent Card | 1% |
| AI usage preferences (Content-Usage) | <1% |
| Markdown version linked from the page | 4% |
| Licensing terms for AI (RSL) | <1% |
| Text-and-data-mining reservation (TDMRep) | <1% |
| WebMCP tools in the page | 1% |
| Buttons, links and fields have names | 53% |
| Claims come with figures, quotes or sources | 44% |
| Your page's HTML isn't oversized | 98% |
| Schema facts also appear in the page text | 57% |
| Description fits a search snippet | 70% |
| Section headings are phrased as questions | 29% |
| Your page arrives quickly | 89% |
| Engines are allowed to quote the page | 98% |
| Links are real links | 89% |
| Verdict: Chef's kiss | 27% |
| Verdict: Cleans the plate | 42% |
| Verdict: Needs seasoning | 16% |
| Verdict: Undercooked | 7% |
| Verdict: Sent back to the kitchen | 0% |
| Turned our scanner away | 8% |
When we changed how we judge
- : Broken links, Safe Browsing and a first group of agent readiness checks
- : Four new checks, and a caching check
- : A site that turns our scanner away now gets a report, not an error screen
- : The verdict now follows how well AI can read you, not just the total
- : The AI crawler list was refreshed, and page weight is measured properly
- : A site that blocks AI crawlers can no longer look healthy
- : A site that blocks our scanner gets no score at all
- : Site scans no longer count the same page more than once
- : A missing llms.txt is reported as a fail again
- : Rebalanced one check that was counting for more than it proved
- : A category we could not measure no longer shows as a zero
- : Every scan now records which version of our scoring produced it
- : We look in more than one place before telling you something is missing
- : Your site score now reflects how much of your site is actually right
- : Your score now counts only what decides whether a machine can read your page
- : "AI access" now means access, and what you tell AI has its own place
- : Site scans now use all the pages they are allowed to read
- : Pages that build themselves in the browser are now read the way a browser reads them
- : A "Robot or human?" screen is no longer scored as your site
- : A paywall or a rate limit is no longer called a firewall
- : A page withheld for legal reasons is no longer called bot protection
- : A directory of AI crawlers, with who runs each one and what it is for
- : A site scan now names the pages your robots.txt disallows for us
- : New checks, and fewer signals only a person can see
- : Short brand names are compared, and checks we could not run say why
- : Five checks stopped misreading a few kinds of page
- : One contact-file check moves to shown, not scored; two others are re-balanced
- : Analytics look-alikes no longer count, and page titles read the way a browser reads them
- : A Content-Signal mentioned in a robots.txt comment no longer counts
- : Placeholder links no longer count as internal links
- : Pages with search boxes and lazy-loaded images are read in full
- : Link checks no longer test a link to the page itself
- : The sitemap your robots.txt points to is now checked
- : Structured data about a page's main subject is now read
- : A site whose secure version answers slowly is no longer reported as not using https
- : A slow secure version no longer marks a site down for its canonical or HSTS
- : Analytics is found wherever it sits on the page, and more tools are recognised
- : The www and non-www check no longer skips when the other name does not exist
- : A link inside a button now names the button
- : A button inside a link now names the link
- : Structured data in menus, headers and footers is read where it belongs
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Better or worse
Not enough history yet.
Fixed means a site failed a check, or only got a “worth a look”, the first time we scanned it and passes it now. Broke means the reverse. Includes our changes means we changed how a check is judged between the two scans. We count those sites apart and never as a site improving.
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Performance
Not enough sites have a performance result yet.
Core Web Vitals are Google's three measures of loading, responsiveness and visual stability. Field results come from real Chrome visitors over the past four weeks. Lab results come from one test run.
| Result | Share of sites |
|---|---|
| All three Core Web Vitals in the good range | — |
| Largest contentful paint in the good range | — |
| Layout shift in the good range | — |
| Interaction to next paint in the good range | — |
| Lab test: good | — |
| Lab test: needs improvement | — |
| Lab test: poor | — |
| Lab and field results agree | — |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Readiness ladder
In October 2026, 21% of the sites we scanned cleared every one of the gates.
The gates are “Automated visitors can reach your pages”, “AI crawlers aren't blocked”, “Reached without a redirect chain”, “Your answer is in the HTML”, “Structured data is present”, “llms.txt is present and valid”: the basics a machine needs before it can use a page.
| Gates cleared | Share of sites |
|---|---|
| No gates cleared | 6% |
| One of six gates cleared | 3% |
| Two of six gates cleared | 3% |
| Three of six gates cleared | 6% |
| Four of six gates cleared | 29% |
| Five of six gates cleared | 33% |
| All six gates cleared | 21% |
Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Embed or share this figure
Method
- A site is a host exactly as we scanned it: the www version and the bare domain are different sites.
- Each site counts once, on its latest scan of the homepage (or its latest scan of any page if it has no homepage scan). A scan that failed outright is never counted.
- "Turned our scanner away" means the site answered our scanner with a refusal, a bot check or an empty page we could not read. A site that was down or timed out is left out, not counted as turning us away.
- The headline figures use scans from the last three months. The month-by-month lines use calendar months in UTC, so a month's figure can differ from the headline one.
- Figures are whole percentages. A share above zero is never rounded down to nothing, and one short of all is never rounded up to all.
- A figure appears only when enough different sites sit behind it. Below that it reads "not enough sites yet", and we never merge views to fill a gap.
- We show statuses and verdict tiers, never score distributions. A site we could not judge on a check is counted as "couldn't judge", not as a failure.
- When we change how a check is judged, it is marked on the change-over-time lines and listed in the changelog.
- Based on the sites Bot Appétit has scanned: ones people chose to scan, plus lists we scan for research. Not a random sample of the web.
Figures generated 2026-10-25 from build 56914ff6afaa82df. Download the data as JSON or CSV; both hold percentages only.
How to cite this
Milk Moon Studio, "Bot Appétit State of the Web & AI Readiness", scan.milkmoonstudio.com/state-of-the-web/, figures from October 2026.
Figures change every hour, so say which month you took them from.
How does your site compare?
A free scan checks all of this on your page in seconds, with no signup.
Want it handled? We do that too. No pitch, just a plain-English chat.
Book a call