Agent-readiness crossed from idea to measurable infrastructure this week. On April 17, as Cloudflare Brokers Week prolonged into its sixth day, the corporate shipped isitagentready.com, a public scanner that scores any web site on how ready it’s for AI brokers. Paste a URL, get a rating, see which checks handed and which failed, learn AI-generated steerage on the way to enhance. For the primary time, the agent-legibility dialog moved from “is my web site prepared for brokers” as a intestine feeling to “my web site scored X out of 100 in these 5 classes, listed here are the failing indicators.”
The Agent Readiness Rating is an actual shift. It is usually a structurally deceptive software for those who cease studying after the composite quantity.
I ran the scan on this web site (nohacks.co) and scored 33 out of 100, Stage 2 “Bot-Conscious.” The robots.txt handed. The sitemap handed. The AI bot guidelines in robots.txt handed. Content material Indicators handed. Then the rating collapsed throughout classes the place a content-only weblog genuinely doesn’t want what the scanner checks for. Extra on that in a minute.
First, the context. Cloudflare has been transport agent-facing infrastructure all week. The Agent Readiness Rating arrived alongside Agent Reminiscence, Shared Dictionaries, Redirects for AI Coaching, an LLM compression method referred to as Unweight, and a feature-flag software referred to as Flagship constructed for AI-generated code. 4 days earlier, they shipped Undertaking Assume (a brand new Brokers SDK), and OpenAI matched it inside hours with their very own Brokers SDK. I wrote about that in The Agent Runtime Wars Began This Week. The readiness scanner is the logical subsequent piece: If runtimes are the brand new browser layer, web site homeowners want a strategy to check whether or not their web site is legible to that layer. Cloudflare shipped the tester.
The query this text solutions is narrower: What does the scanner really test, what do you have to do along with your rating, and the place is the scoring structurally deceptive sufficient that the quantity by itself leads you astray?
What Cloudflare Shipped: Scanner, API, And An MCP Endpoint Brokers Can Name On You
The scanner is at isitagentready.com. Paste any URL, decide an internet site sort (All Checks, Content material Website, or API/Software) to scope which indicators get scanned, hit Scan. The scanner fetches the homepage and a handful of well-known paths, runs a set of checks towards every, and returns a scored report with go/fail markers, standing codes, response our bodies, and AI-generated steerage on what to repair.
The scanner can be out there in three different methods:
- Built-in into Cloudflare Radar, so the identical checks run alongside Radar’s present URL evaluation.
- Uncovered programmatically by way of the Cloudflare URL Scanner API for automation.
- Obtainable as a stateless MCP server at
/.well-known/mcp.json, so any MCP-compatible agent can name the scan as a software and motive over the outcome
That final one is price sitting with for a second. Cloudflare shipped an agent-readiness scanner that brokers themselves can name to audit web sites earlier than deciding the way to work together with them. The scanner checks whether or not your web site is prepared for brokers, and any agent can invoke it to determine the way to work together with you earlier than arriving. The measurement and the measured are beginning to share the identical floor.
Again to the sensible query. What precisely does it test?
16 Checks, 5 Classes: What The Scanner Truly Exams
The scanner teams its checks into 5 classes. Here’s what each seems to be for, grouped by what the test really means in follow.
Discoverability (3 Checks)
Whether or not the web site publishes the essential metadata an agent wants to seek out what’s the place.
- robots.txt exists. The traditional crawl-policy file. An agent that follows robots.txt wants it to exist and parse.
- sitemap.xml exists. Both declared by way of a Sitemap directive in robots.txt or out there at the usual path. An agent that wishes to enumerate pages makes use of the sitemap.
- Hyperlink headers (RFC 8288). HTTP Hyperlink headers pointing to canonical, alternate, or associated sources. Helpful for brokers that parse responses fairly than HTML.
Content material (1 Test)
- Markdown for Brokers. Content material negotiation. The scanner sends
Settle for: textual content/markdownand checks whether or not the web site returns Markdown as an alternative of HTML. That is Cloudflare’s personal proposal fairly than an IETF spec, although the mechanism (HTTP content material negotiation by way of theSettle forheader) is commonplace. Actual agent runtimes desire Markdown as a result of it’s cheaper to tokenize and simpler to parse than HTML. Some early movers (Cloudflare itself, a handful of docs web sites) assist Markdown content material negotiation; most web sites don’t.
Bot Entry Management (3 Checks)
- AI bot guidelines in robots.txt (RFC 9309). Whether or not robots.txt incorporates directives for AI-specific person brokers (GPTBot, ClaudeBot, PerplexityBot, and so forth.).
- Content material Indicators in robots.txt. An rising spec for expressing per-URL entry guidelines inside robots.txt. Parsed as
Person-agent: *adopted byContent material-signal:directives. Adoption is minimal proper now. - Internet Bot Auth request signing. HTTP message signatures at
/.well-known/http-message-signatures-directorythat permit brokers show their identification cryptographically. That is the Agent Identify Service aspect of issues, Cloudflare shipped with GoDaddy earlier in Brokers Week. Adoption is nearly zero outdoors Cloudflare’s personal properties.
API, Auth, MCP & Ability Discovery (6 Checks)
- API Catalog (RFC 9727). A machine-readable index of an internet site’s API endpoints at
/.well-known/api-catalog. - OAuth / OIDC discovery (RFC 8414). Customary OAuth 2.0 authorization server metadata at
/.well-known/oauth-authorization-serverand/.well-known/openid-configuration. - OAuth Protected Useful resource (RFC 9728). A web site declaring which endpoints are OAuth-protected and the way to authenticate.
- MCP Server Card (SEP-1649). A Mannequin Context Protocol server promoting its capabilities at
/.well-known/mcp/server-card.json. SEP-1649 is a draft proposal contained in the MCP spec course of. - Agent Expertise index. An inventory of agent-callable expertise at
/.well-known/agent-skills/index.json. Additionally rising. - WebMCP (Experimental). An in-page JavaScript API registering agent-callable instruments by way of
navigator.modelContext. The scanner makes use of headless browser rendering to detect whether or not the web site registers any WebMCP instruments on web page load.
Commerce (3 Non-obligatory Checks, Not Scored On Non-Commerce Web sites)
- x402 cost protocol. HTTP 402 Cost Required infrastructure for agent-native funds.
- UCP profile (Common Commerce Protocol). Google’s merchant-metadata commonplace at
/.well-known/ucp. - ACP discovery doc (Agentic Commerce Protocol). At
/.well-known/acp.json.
The Commerce class is flagged “non-compulsory” on non-commerce web sites. The scanner detects whether or not any ecommerce indicators are current and, if not, shows the commerce checks for informational functions with out counting them within the rating.
That final design element issues. It’s proof Cloudflare anticipated precisely the issue the remainder of this text is about.
Nohacks.co Scored 33/100, Stage 2 Bot-Conscious
I ran the scan on nohacks.co. The outcome was 33 out of 100, Stage 2 “Bot-Conscious.”
A observe on that quantity: After the primary scan, I added Content material Indicators directives to robots.txt, which moved Bot Entry Management from 50 to 100 and pulled the composite up eight factors from an preliminary 25. Each different class beneath is unchanged from the primary scan. I’ll come again to the Content material Indicators repair and why I made it on the finish of this part.
Here’s what drove every class rating:
- Discoverability: 67. robots.txt and sitemap.xml handed. Hyperlink headers failed as a result of this web site doesn’t emit
Hyperlink:headers in its responses. - Content material: 0. Markdown content material negotiation isn’t configured. The web site returns HTML whatever the
Settle forheader. - Bot Entry Management: 100. Each scored checks handed. AI bot guidelines in robots.txt (I’ve express guidelines for AI person brokers) and Content material Indicators in robots.txt (I added these after the primary scan). Internet Bot Auth request signing is listed on this class as an informational test, however not counted towards the two/2.
- API, Auth, MCP & Ability Discovery: 0. All six checks failed. No API Catalog. No OAuth discovery. No OAuth Protected Useful resource metadata. No MCP Server Card. No Agent Expertise index. No WebMCP instruments on the web page.
- Commerce: not scored. nohacks.co has no e-commerce. The Commerce checks all failed, however the class is accurately excluded from the composite rating.
That could be a 33 on a scanner constructed by the corporate I most belief to grasp the place the agent-ready internet goes. I take into account this web site moderately well-designed for brokers. The robots.txt is clear and express. The content material is server-rendered, machine-readable HTML with clear semantic construction. The sitemap is present. The URLs are steady. If you happen to requested me per week in the past whether or not this web site was agent-ready, my reply could be someplace between “largely sure” and “for what it must do, sure.”
And but: 33, Stage 2.
The scanner is measuring what it says it’s measuring. The composite rating, by itself, continues to be the improper quantity to optimize for.
One observe on the Content material Indicators repair, as a result of it’s related to the Goodhart argument later on this article. Content material Indicators is a Cloudflare proposal with nearly no deployment past Cloudflare-aligned crawlers. I debated including it for precisely the score-chasing motive this text warns about. I made a decision it was defensible for 2 causes. First, the repair is declarative, not ornamental. The directives state actual coverage about what ought to occur with my content material, and the assertion has which means even when the spec fails. That’s completely different from including an empty MCP Server Card to fulfill a scorer. Second, for an internet site that writes about agent-readiness particularly, publicly declaring content material coverage is editorial follow no matter which crawler respects it. The repair was one decide to public/robots.txt and the directives are readable by any human curious sufficient to test.
Identical Web site Scores 33 Or 67 Relying On The Preset You Choose
On the All Checks preset, nohacks.co scores 33 out of 100, Stage 2 “Bot-Conscious.” On the Content material Website preset, similar web site, similar day, completely different scan configuration, it scores 67, nonetheless Stage 2 “Bot-Conscious.” Practically double the composite quantity. The 34-point hole is the distinction between two scan configurations of the identical scanner, not a distinction between two web sites.
Here’s what the Content material Website preset modifications within the scan configuration:
Working that preset on nohacks.co produced this outcome:
4 of six scored checks go. The 2 failures are unambiguous remediation targets: Hyperlink headers by way of HTTP response configuration, Markdown content material negotiation by way of origin or CDN response logic. Each ship towards actual agent-runtime conduct right now. Neither is a proposal-stage format that can solely perhaps grow to be a regular. That is the sincere studying of nohacks.co’s agent-readiness state: two particular, actionable gaps.
The Appropriate Toggle Is Hidden, And The Default Rating Is Flawed
The scanner is doing its job. It is aware of a weblog doesn’t want an MCP Server Card. It is aware of a podcast archive doesn’t publish an API catalog. The Content material Website preset isn’t beauty. It removes irrelevant checks and offers a content material web site an correct studying towards requirements that really apply.
The issue is that the preset is hidden. When a person lands on isitagentready.com and pastes a URL, the default scan is All Checks. The Website Kind toggle that may swap to Content material Website or API/Software lives inside a Customise dropdown that the majority customers won’t ever open. The person clicks Scan, reads the composite quantity, takes a screenshot, shares it. The shareable quantity, the one which travels on social media, the one opponents evaluate throughout, is the All Checks composite.
For a content material web site that runs the default scan with out studying particular person checks, the composite is structurally too low. The 33 on nohacks.co is improper for the type of web site nohacks.co is. The 67 from the Content material Website preset is the correct studying. Two numbers from the identical scanner on the identical web site. The correct quantity is behind a dropdown. The improper quantity is on the entrance web page.
Any internet skilled who runs the scanner and plans to share the rating anyplace public must open Customise, choose the preset that matches their web site sort, and re-run earlier than sharing. With out that step, the general public rating will understate the web site’s precise agent-readiness, and the hole between the shared quantity and the correct quantity will likely be bigger for content material web sites than for API web sites (that are nearer to the All Checks baseline). Learn the person checks. Don’t share a composite till you realize which preset produced it.
For the file: the 67 is bothering me. I’m going to go get the 100. I do know precisely what the Goodhart part beneath is about to warn towards, and I’m going to do it anyway. Two fixes stand between me and the 100. Each are five-minute jobs. Each map to actual agent-runtime conduct (Hyperlink headers for discovery, Markdown content material negotiation for environment friendly agent parsing), so at the very least the motivation is authentic and never pure score-chasing. That caveat can be precisely what score-chasers say. Public scores are a gravitational subject. Even the individual writing a protracted article about their unreliability finally ends up orbiting.
Agent Readiness Measures Supply, Not Message
Each class the Agent Readiness scanner checks is about supply: discoverability, content material negotiation, bot entry, API discovery, commerce protocols. None checks the standard of the message itself.
The scanner by no means asks whether or not your headlines are clear, whether or not your product descriptions persuade, whether or not your content material solutions the question properly, whether or not your writing is any good. These are search engine optimisation and CRO questions. They occupy the self-discipline of creating the message higher. The Agent Readiness Rating occupies a unique self-discipline totally. It asks whether or not an agent can fetch your content material, parse the format it arrives in, authenticate towards your endpoints, name your capabilities, pay on your outputs.
That’s the distinction that issues. Classical internet optimization (search engine optimisation, CRO) is about what you say and the way persuasively you say it. Agent-readiness is about the way you ship what you say to a non-human reader. Two web sites can publish word-for-word similar content material. One serves it as server-rendered HTML with semantic markup, responds to Settle for: textual content/markdown, exposes structured knowledge, returns predictable response codes. The opposite serves it as a JavaScript-rendered single-page utility with no content material negotiation and an inconsistent error floor. The message is similar. The supply is completely different. The agent-readiness rating will likely be completely different. And it will likely be proper to be completely different, as a result of the supply is what the agent interacts with.
That is additionally why agent-readiness fixes are typically orthogonal to search engine optimisation and CRO work. You may enhance an agent-readiness rating with out rewriting a single phrase of your content material. You may also have world-class search engine optimisation content material that scores a ten on the agent-readiness scanner as a result of none of your supply pipeline was designed for machine customers. search engine optimisation and CRO work on the content material layer. Agent-readiness works on the transport and protocol layer. They’re adjoining however not the identical craft, and treating them as the identical is the error that turns an agent-readiness undertaking right into a content-rewrite undertaking and misses the precise repair.
The individuals who will do properly over the following a number of years are those who cease arguing about which self-discipline issues extra and begin recognizing they occupy completely different layers of the stack.
3 Goodhart Dangers Constructed Into The Agent Readiness Rating
Goodhart’s legislation says that when a measure turns into a goal, it stops being a great measure. The Agent Readiness Rating is well-designed, however additionally it is now a public, shareable, in contrast quantity, which produces three predictable behavioral failures within the wild.
The primary threat is that web site homeowners will optimize for the quantity fairly than for actual agent conduct. Add an MCP Server Card that factors nowhere as a result of the scanner needs one. Publish an Agent Expertise index with no precise expertise. Ship a WebMCP software that does nothing simply to go the detection test. The rating goes up, and nothing modifications for actual agent runtimes visiting the web site.
The second threat is that consultancies will begin promoting “Agent Readiness Rating optimization” as a service, promoting the rating fairly than the underlying structure. The historical past of search engine optimisation offers us a century of information on how this performs out. PageRank grew to become a goal, and a decade of link-spam financial system grew up round it. Core Internet Vitals grew to become a goal, and a era of performance-theater optimizations adopted. The Agent Readiness Rating is a better-designed metric than both of these had been at launch, however the identical gravity applies.
The third threat is that the scanner’s inclusion of rising requirements as scored indicators will speed up the adoption of these requirements previous the purpose the place they’re prepared to hold actual visitors. The scanner checks for llms.txt, a proposed format for exposing web site content material to language fashions. Llms.txt isn’t a ratified commonplace, has no governing physique, and has competing proposals for the way it must be structured. Together with it as a scored sign offers it weight it has not earned within the ecosystem. A web site proprietor trying to repair a failing test is the marginal adopter who suggestions a proposal right into a de facto commonplace earlier than the spec work is finished.
None of those failure modes are hypothetical. They’re how each public measurement rating within the historical past of the net has performed out. The Agent Readiness Rating is best than most as a result of Cloudflare is sincere about what it’s, as a result of the per-check element is obtainable proper alongside the composite quantity, and since the Commerce class accurately excludes itself on non-commerce web sites. That honesty is a function price defending. Web site homeowners and the consultancy business will likely be tempted to deal with the composite quantity because the goal anyway.
Don’t do that.
6 Weekend Fixes That Map To Actual Agent Runtimes
Six actions for an internet skilled working the scanner the weekend of its launch, ordered from highest-leverage to lowest:
- Run the scan in your web site. It takes about 30 seconds. Word the rating and open the detailed report. The element is the place the sign is.
- Repair the failing checks that ship towards actual agent runtimes right now. These are those whose absence measurably hurts your web site for brokers visiting it proper now:
- robots.txt. If lacking, add one. If current, be certain that it incorporates particular guidelines for AI person brokers (GPTBot, ClaudeBot, PerplexityBot, Google-Prolonged, and so forth.).
- sitemap.xml. If lacking, generate one and hyperlink it from robots.txt. Maintain it present.
- Markdown content material negotiation. Configure your origin or CDN to return
textual content/markdownwhen theSettle forheader requests it. Cloudflare’s personal AI Crawl Management has first-class assist for this. Different suppliers require customized server logic. - Structured knowledge. Ship schema.org JSON-LD for the content material varieties your web site publishes (Article, Product, Group, BreadcrumbList). This isn’t a scored test, however it’s the highest-leverage repair for quotation conduct throughout each agent runtime at the moment deployed.
- Deal with the proposal-stage codecs as a watch listing, not a guidelines. llms.txt, Content material Indicators in robots.txt, Internet Bot Auth, API Catalog, MCP Server Card, Agent Expertise, WebMCP, ACP, UCP are all actual working requirements in some sense. They aren’t transport towards actual agent-runtime conduct at scale but. Watch them. Implement them when your stack has a motive to, not as a result of the scanner flags them.
- Ignore the composite quantity in your personal monitoring. Monitor particular person test outcomes over time. A web site that goes from 3 of 5 real-runtime checks passing to five of 5 has measurably improved, even when the composite rating barely moved as a result of the ten proposal-stage checks nonetheless fail.
- Re-scan after modifications. The scanner is quick, free, and out there by way of the URL Scanner API if you wish to script regression checks into your deployment pipeline.
- Skip the consultancies promoting Agent Readiness Rating optimization. The work is easy sufficient {that a} half-day audit and a centered remediation dash will beat any packaged service.
The scanner is the software. The work continues to be the work.
Vendor-Particular Scanners Are Coming: Monitor What Each Scanner Exams
The Agent Readiness scanner is standards-list-shaped: a set of checks towards a hard and fast listing of protocols and codecs, some ratified (RFC 8288 Hyperlink headers, RFC 9309 robots.txt guidelines, RFC 8414 OAuth discovery, RFC 9727 API Catalog, RFC 9728 OAuth Protected Useful resource), some rising proposals (MCP SEP-1649, WebMCP, Content material Indicators, Internet Bot Auth, x402, UCP, ACP, llms.txt). The following factor that occurs within the ecosystem is predictable: Different distributors will ship their very own scanners towards their very own most well-liked lists. The overlap will likely be vital as a result of a lot of the ratified requirements are uncontroversial. The divergence will likely be through which proposals every vendor scores for.
That divergence is the place the agent-readiness measurement story will get attention-grabbing. A Cloudflare scanner that checks for Internet Bot Auth and UCP is betting. A Google scanner, if it ships, would test for a number of the similar issues and a few completely different ones (Google has UCP, doesn’t have Internet Bot Auth). A Perplexity scanner would test for one more set. Web site homeowners would see completely different scores from completely different scanners on the identical web site. The composite quantity, already not reliable, turns into vendor-specific.
The sign price monitoring is which checks present up in each scanner that ships. These are the de facto requirements. The checks that solely present up in Cloudflare’s scanner are Cloudflare’s bets. Some will win. Most won’t.
That is the sample that made me comfy publishing an article a few Cloudflare software on the day it shipped. The Agent Readiness Rating is actual. The thesis behind it (agent-readiness is a measurable property) is the suitable thesis. The particular scorecard is model one in all one thing that’s going to have dozens of variations, every reflecting its vendor’s bets. Internet professionals ought to have interaction with the version-one scorecard, repair what it accurately flags as actual, watch what it flags as rising, and preserve their very own working listing of which checks survive throughout each scanner that ships within the subsequent six months.
That working listing is the actual agent-readiness commonplace. The composite rating is the advertising layer.
Run the scan. Learn the report. Repair what issues. Watch what would possibly.
Extra Assets:
This put up was initially printed on No Hacks.
Featured Picture: RobinRmD/Shutterstock
