How agent-ready is the average site?
Across 2,337 distinct sites, the mean agent-readability score is 56 out of 100 and the median is 55. The score is the share of applicable checks a site passes; see the methodology for the formula and rating bands.
- Excellent9% · 215
- Good8% · 187
- Fair47% · 1,090
- Needs improvement36% · 845
| Rating band | Sites | Share |
|---|---|---|
| Excellent (90–100) | 215 | 9% |
| Good (70–89) | 187 | 8% |
| Fair (50–69) | 1,090 | 47% |
| Needs improvement (0–49) | 845 | 36% |
How many sites publish an llms.txt or AGENTS.md?
The discovery files are the cheapest agent-readiness win and the most-skipped. In this corpus, 58% of sites publish a valid llms.txt, 40% ship the llms-full.txt companion, and 37% publish an AGENTS.md. The mean llms.txt sub-score is 49/100.
| Discovery file | Sites | Adoption |
|---|---|---|
| Publish a valid llms.txt | 1,363 | 58% |
| Publish llms-full.txt | 933 | 40% |
| Publish AGENTS.md | 866 | 37% |
What are the most common agent-readability failures?
These are the site-level and llms.txt checks that fail most often, ranked by fail rate among the sites the check actually ran on. They’re the highest-leverage fixes for a typical site.
| Check | Fail rate | Sites checked |
|---|---|---|
| S13AGENTS.md has required sections | 86% | 2,337 |
| S11sitemap.md has headings + links | 84% | 2,337 |
| S10sitemap.md exists | 66% | 2,337 |
| S12AGENTS.md exists | 63% | 2,337 |
| L4Blockquote summary | 62% | 2,337 |
| L5H2 file-list sections | 60% | 2,337 |
| L6Link format correct | 60% | 2,337 |
| L10llms-full.txt available | 60% | 2,337 |
| S2llms.txt Content-Type | 58% | 2,337 |
| L9Content-Type: text/plain | 58% | 2,337 |
Which agent protocols are sites actually adopting?
Agent-protocol manifests are conditional - a site only “has” one if it advertises the matching well-known endpoint. Adoption is still early, and the counts are small, so these figures lead with the raw number of sites.
| Protocol surface | Sites | Share |
|---|---|---|
| API catalog | 281 | 12% |
| MCP server card | 236 | 10% |
| Agent Skills discovery | 224 | 10% |
| A2A agent card | 219 | 9% |
| Web Bot Auth | 151 | 6% |
| agent-permissions.json | 136 | 6% |
| agents.json | 134 | 6% |
| UCP profile | 127 | 5% |
| NLWeb endpoint | 5 | 0% |
| x402 payments † | 3 | 0% |
† Fewer than 5 sites — indicative only, not a robust rate.
How many sites block AI crawlers?
A robots.txt that disallows AI user-agents keeps a site out of AI search and assistants entirely. 6% of sites in this corpus block at least one major AI crawler. The most-blocked crawlers:
| AI crawler | Sites blocking it | Share |
|---|---|---|
| PerplexityBot | 105 | 4% |
| YouBot | 99 | 4% |
| OAI-SearchBot | 86 | 4% |
| Meta-WebIndexer | 86 | 4% |
| DuckAssistBot | 86 | 4% |
| Claude-SearchBot | 85 | 4% |
| Amzn-SearchBot | 82 | 4% |
| MistralAI-Index | 79 | 3% |
| GPTBot | 27 | 1% |
| Google-Extended | 25 | 1% |
How was this measured?
Every figure on this page is computed from 2,337 distinct sites that were scanned on agent-ready.dev between 2026-04-18 and 2026-10-19. To stop a single heavily-scanned site from skewing the corpus, we keep only the most recent completed scan per domain, and we exclude sites operated by Agent Ready so our own pages don’t flatter the numbers. No individual site is named - the report is aggregate and anonymous by design.
These figures recompute daily, so the exact number you read today will have moved by next week. For a stable reference, cite the frozen 2026 edition — a point-in-time snapshot whose numbers never change.
This is a sample of sites whose owners chose to check their agent readiness, so adoption rates here run ahead of the web at large. Read it as a trend signal, not a census. The full check registry and scoring rules live on the methodology and specs pages.
Frequently asked questions
- Where does this data come from?
- Every public scan run on agent-ready.dev is stored, and this page aggregates them. To avoid letting a single heavily-scanned site skew the numbers, we take one scan per domain - the most recent completed scan - and compute statistics across those distinct domains. No individual domain is named anywhere on this page; the report is deliberately aggregate and anonymous.
- Is this a representative sample of the whole web?
- No. It's a sample of sites that someone chose to scan with Agent Ready, which skews toward sites whose owners already care about AI agents - so adoption rates here are almost certainly higher than the web at large. Read the numbers as 'among sites being actively checked for agent readiness', not as a census. The sample size and date range are shown at the top of the page so you can judge the weight of each figure.
- How is the agent-readability score calculated?
- Each scan runs dozens of checks across four layers - site-level discovery files, per-page extraction signals, the llms.txt standard, and agent-protocol manifests - and the Vercel score is the share of applicable checks that pass, expressed 0–100. A separate accessibility sub-score (WCAG 2.2 and layout stability) is also reported per scan but is graded on its own and isn't folded into the Vercel score. The full scoring formula, rating bands, and per-check weighting are documented on the methodology page.
- How often do these numbers change?
- The page recomputes from the live corpus on a daily cycle, so the figures drift as more sites are scanned and as scanned sites improve. The 'as of' date at the top reflects the latest scan included. For a fixed point-in-time reference, cite the date alongside the statistic.
- Which checks count as the 'most common failures'?
- We rank the site-level and llms.txt checks that run on essentially every scan by how often they fail, and exclude any check that ran on too small a slice of the corpus to be meaningful. Conditional agent-protocol checks (MCP, A2A, x402, and the rest) aren't in the failure ranking - a site isn't 'failing' a protocol it never claimed to support - so those appear in the adoption section instead.