On April 27, 2026, Cloudflare Radar recorded something that wasn’t supposed to happen for another 18 months: bot traffic crossed 57% of global HTML content HTTP requests, surpassing human traffic for the first time. Cloudflare CEO Matthew Prince had put the milestone in late 2027. Agentic AI pulled it forward.
This measures HTML requests only. Video streaming, email, gaming – these are excluded. So it’s specifically the “readable web,” the part built for eyes to scan and brains to parse. Humans still dominate plenty of the internet by bytes. But the part that carries words and links? Bots win.
The image that stuck with me: a human shopping for a camera visits maybe five or six sites. An AI agent visits thousands.
llms.txt: Writing for the Agent at the Door
llms.txt is a proposed standard – a markdown file at the root of your site that tells AI agents what to read and where to go. Think robots.txt, but for LLM crawlers instead of search bots. SE Ranking surveyed 300,000 domains and found about 10% adoption. Early days.
This isn’t really an SEO play. It’s B2A – Business to Agent. Coding tools like Cursor and Claude Code already fetch llms.txt from documentation sites as a routine step. They’re looking for a signal that says “here’s the important stuff, skip the nav.”
Netlify CEO Mathias Biilmann coined “Agent Experience” (AX) in January 2025. UX came first, then DX as developers became first-class users. AX follows the same arc. Once someone names a thing, the concept has arrived.
AXI: Designing CLIs That Don’t Waste Tokens
AXI (Agent eXperience Interface) is 10 design principles for building agent-friendly command-line tools. If you’ve ever wondered why your MCP integration feels bloated, the numbers explain it: MCP burns 185,000 input tokens on schemas for just 30 tools. A plain CLI doing the same work uses 79,000. That’s 2.3x overhead before your agent does anything useful.
A format called TOON cuts tokens another 40% versus JSON. Anthropic acknowledged the same problem with their Tool Search Tool on the Claude API, which defers tool definitions and lazy-loads them on demand. Their engineering write-up reports 85% reduction in tool token overhead. It’s been the default in Claude Code since January 2026.
Benchmarked over 490 runs: AXI hit 100% task success at $0.074 per task, 21.5 seconds, 4.5 turns. MCP cost $0.100. Plain CLI cost $0.088. AXI beat both on every metric.
The insight that landed hardest for me: principled design wins. CLI versus MCP is a distraction. Human niceties – colored output, table alignment, paragraphs of help text – are token waste when your reader is an agent. The agent doesn’t care that your table is pretty. It cares that the data is structured and the error messages are actionable.
I Stopped Writing Docs for People
I work at a Korean company. My internal documentation is in English. Not translated from Korean – English from the first keystroke.
This started as a practical decision. With OpenAI’s cl100k_base tokenizer, the same content in Korean takes on average 2.36x the tokens of English. A NeurIPS 2023 paper put the upper bound at 15x across languages for equivalent text. That difference maps directly to API cost and context window consumption. The same context budget fits more than twice as much English documentation. When your AI assistant is summarizing a design doc or retrieving a decision record, that matters.
The practical decision revealed something bigger. I don’t really write internal docs for my teammates anymore. I write them for the AI that my teammates will query. The bar used to be: “can a new person read this and understand the decision?” Now it’s: “can the AI surface this when someone asks the right question?”
My teammates still read the docs. They mostly read them through an AI that retrieved the relevant sections and explained the context. The doc is still for humans – but the first reader is the model.
So these days I try to leave raw material instead of finished answers. A doc that only records the conclusion can’t be applied to a question nobody anticipated. I write down the decision along with the reasoning, the constraints, and the assumptions we held at the time. Retrieval pulls fragments, not whole pages, so each section has to stand on its own – and “recently” gets replaced with an actual date.
Deleting matters as much as updating. A human reading a stale doc gets suspicious; an AI that retrieves one cites it with full confidence. Stale documentation doesn’t mean “no answer” – it means a confidently wrong one. Two contradicting versions are worse than none.
Maybe that’s a strange way to think about it. It just works better.
What This Actually Changes
The honest version: llms.txt at 10% adoption is early. It’s still an informal proposal. Standards could fragment. Nobody knows how long AXI design principles will take to spread. The term “AX” might stay niche.
The underlying shift doesn’t depend on any of those things settling. Bots already read more of the readable web than humans do. That happened. The question is whether you account for it in what you build.
When DX followed UX, human users didn’t disappear. We added a reader. The same thing is happening now. The README I write, the CLI output I design, the internal wiki page I draft at 11pm – the first reader of all of it may no longer be a person.
That feels a little strange. And more natural than I expected.
References
- Cloudflare Radar – bot vs. human traffic data
- Bots have now passed human traffic online – Tom’s Hardware – Matthew Prince on the milestone arriving 18 months early
- llms.txt Explained: The Honest Guide – Codersera – includes SE Ranking’s 300K-domain adoption survey
- Agent Experience (AX) – Netlify’s AX concept hub
- AXI: Agent eXperience Interface – 10 design principles and benchmarks for agent-friendly CLIs
- Tool Search Tool – Claude API Docs – defer_loading-based lazy loading
- Introducing advanced tool use – Anthropic Engineering – measured 85% reduction in tool token overhead
- Anthropic brings MCP tool search to Claude Code – Tessl – default in Claude Code
- Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) – up to 15x tokenization gap across languages
- Working with CJK text in Generative AI pipelines – Anthony Shaw – Korean at 2.36x English tokens