The Agentic Web: Architecting for AI Agents, Not Just Users

Q: What is the agentic web and how is it different from SEO?

The agentic web is the reality that a large and growing share of website traffic comes from machines — crawlers building model indexes, retrieval bots answering live user questions, and agents acting on a user’s behalf — rather than from people with browsers, and that these visitors interact with your site in fundamentally different ways than humans do. The difference from SEO is one of objective rather than technique. Traditional SEO optimised for a ranking position that produced a click: you wanted to appear high in a results page so a human would visit your site, and success was measured in sessions. Agentic-web optimisation aims for something different: being extractable, quotable, and actionable by a machine that may never send you a click at all. When someone asks an assistant a question in your domain, the answer is assembled from whatever content the model could actually parse and cite, and your influence flows through that citation rather than through a visit. The practical implications diverge from classic SEO in specific ways: machines do not see visual hierarchy, do not wait for client-side rendering, do not scroll, and do not respond to persuasion, so hero animations and CTA placement are irrelevant to them while semantic structure, server-rendered content, and unambiguously-stated claims are decisive. There is also an entirely new category SEO never addressed: acting agents that want to transact — book, buy, compare, submit — which need stable form semantics or, better, a real API surface. Crucially, the two disciplines mostly do not conflict: a page structured well enough for clean machine extraction is usually clearer for humans too. But agentic readability has to be designed for deliberately, because a site optimised purely for human eyes fails these visitors silently, with no bounce, no error, and no signal in your analytics.

Q: How do I make my website readable and citable by AI agents?

Six changes, in rough order of return. First, server-render the content that matters: if your substance only exists after JavaScript executes, you are betting that every parser is a full browser, and many are not — server-rendered HTML is machine-readable by construction, which is one of the strongest practical arguments for hypermedia or server-component architectures. Second, write answer-first: lead each section with the claim and support it afterwards, because an answer bot pulling a citation needs the fact stated plainly and early, and a page that builds suspense before its conclusion gets skipped in favour of one that does not. Third, add honest structured data: schema.org markup (Article, FAQPage, Product, Organization, HowTo) tells a machine what your content is rather than making it infer, and it should describe what is genuinely on the page — marking up content that is not there is both a policy violation and a trust failure. Fourth, make sections self-contained: an extractable section survives being lifted out of the page, meaning it names its subject rather than saying "as mentioned above" and does not depend on a table three screens up. Fifth, publish machine-facing entry points: a clean sitemap, deliberate robots directives, and increasingly an llms.txt file that points models at your canonical structured content rather than making them guess. Sixth, decide your bot policy deliberately rather than inheriting a template — training crawlers, answer bots, and acting agents are separable populations with different business implications. Two things to avoid: never cloak (serving bots different content than humans is a policy violation and breaks the trust model that makes citation valuable), and do not treat agent traffic as free, because agents hit endpoints far harder than humans and need rate limiting, caching, and abuse controls.

Q: What is llms.txt and do I need one?

llms.txt is a proposed convention — a plain-text file at your site root — that gives language models a curated, machine-friendly map of your site: what it is about, and which canonical pages contain your substantive content, typically in a clean linkable format. Think of it as a counterpart to robots.txt and sitemap.xml aimed specifically at LLM consumption: where robots.txt says what may be crawled and sitemap.xml enumerates URLs for search indexing, llms.txt is about giving a model the context and pointers it needs to understand and use your content correctly, rather than leaving it to infer structure from navigation and heuristics. Do you need one? It is low cost and forward-looking rather than mandatory. It is a proposal with growing but not universal adoption, so the honest framing is that it costs very little to publish and may improve how models understand your site, while not being a substitute for the fundamentals. Prioritise it accordingly: server-rendered content, answer-first writing, semantic HTML, honest schema.org markup, and a clean sitemap all deliver more value today, because they affect every machine visitor regardless of whether it looks for llms.txt. Once those are in place, adding llms.txt is a sensible next step, particularly for content-heavy sites, documentation, and businesses that actively want to be cited by answer engines. Practical guidance if you publish one: keep it genuinely curated rather than an exhaustive URL dump (the value is in pointing at your best canonical content, not reproducing the sitemap), keep it current as your content changes, and make sure the pages it points to are themselves clean, server-rendered, and well-structured — pointing a model at content it then cannot parse defeats the purpose entirely.

Q: Should I block AI crawlers from my website?

It is a business decision that deserves deliberate thought rather than a default, and the key insight is that "AI crawlers" are not one population — training crawlers, retrieval/answer bots, and acting agents have different implications, and blanket allow-everything or block-everything answers are usually both wrong. Separate the cases. Training crawlers gather corpus for model training: blocking them protects your content from being absorbed into a model’s weights without compensation, which some publishers reasonably want, but it does not affect whether you get cited in live answers and it forgoes any long-term influence on what models know about your domain. Retrieval and answer bots fetch your content to answer a user’s question right now, usually with attribution: for most businesses these are valuable and blocking them removes you from the answers your potential customers are receiving — this is the population that most directly affects whether people hear about you at all. Acting agents attempt to do things on your site on a user’s behalf: whether you welcome them depends on your business (an e-commerce site may want agent-driven purchases; a site with abuse-sensitive flows may not), and the right answer is often to channel them into a real API rather than have them drive your UI. The practical posture most businesses land on: allow retrieval and answer bots because citation drives awareness and eventual revenue; take a considered position on training crawlers based on whether your content is your product; and decide about acting agents based on whether agent transactions are desirable, providing a proper interface if they are. Whatever you choose, implement it explicitly in robots directives rather than inheriting a template, review it as the landscape changes, and pair it with rate limiting and caching, because permission and capacity are separate questions.

Q: How do I measure success if AI answers do not send clicks?

By adding metrics that capture influence rather than only visits, because click-through rate now systematically understates your reach — a page can be enormously influential, cited in thousands of AI answers, and show flat traffic. Four measurement changes matter. First, track citations and mentions: periodically query the major assistants with the questions your content should answer and record whether you are cited, which is laborious to do manually but is the most direct signal of agentic-web performance, and increasingly there are tools that monitor brand mentions in AI answers. Second, segment bot traffic from human traffic in analytics — most teams have never separated these, so their dashboards silently mix two populations with completely different behaviour, and until you split them you are optimising against noise (our own analytics show bot-heavy geographies inflating raw totals while the qualified human line moves independently). Third, watch downstream and indirect signals: branded search volume and direct visits tend to rise when models begin recommending you, because users hear your name in an answer and then come looking for you — that lag between citation and visit is invisible if you only measure last-click. Fourth, keep measuring the traditional funnel for the traffic that does arrive, since organic clicks still matter and conversion from them is still how the business runs. The mental model that helps: treat AI citation as brand and awareness spend that happens to be earned rather than paid — you would not expect a billboard to produce attributable clicks, but you would expect it to lift branded search and direct traffic, and citation behaves similarly. Teams that keep judging content solely by sessions will conclude that agentic-web work does not pay; teams that instrument citation and segment their traffic can see the influence they are actually earning.

Q: Do AI agents need an API, or is my website enough?

If agents transacting with your business matters commercially, give them a real interface — an API or an MCP server — because agents driving your human UI is fragile for them and risky for you, and the difference becomes stark as agent traffic grows. Consider what happens without one. An acting agent attempting to complete a task on your site has to interpret your visual interface: find the right form, understand the fields, handle validation errors written for humans, cope with modals and multi-step flows, and re-learn all of it when you redesign. It works, unreliably, and every UI change breaks it silently. From your side, UI-driving agents are hard to distinguish from abuse, hard to rate-limit sensibly, and hard to authenticate properly. With a defined interface, both sides improve: the agent gets stable semantics, explicit capabilities, structured errors, and predictable behaviour, while you get authentication, scoped authorization, rate limiting, auditability, and the ability to evolve your UI freely. The Model Context Protocol has emerged as a standard way to expose capabilities to AI agents specifically, and it is worth evaluating alongside a conventional REST or GraphQL API depending on your consumers. Two caveats. First, the same agent-facing surface is an attack surface: every capability you expose needs scoped, least-privilege authorization and audit logging, because an agent that can act on your system can be driven by an attacker as easily as by a customer — this is the tool-authorization discipline that governs any agent interface. Second, do not skip the fundamentals: even with an API, your public content still needs to be server-rendered, well-structured, and citable, because discovery happens through content long before any transaction happens through an interface. Content makes you findable; the API makes you transactable.

Q: Does building for AI agents hurt the human experience?

Almost never — and where the two appear to conflict, the conflict usually reveals that the human experience was weaker than it looked. The reason is that the things machines need are largely things that make content better for people too. Server-rendered content loads faster and works on poor connections and low-end devices, which is a direct human win. Answer-first writing respects readers’ time by stating the conclusion before the supporting detail, which is exactly what busy readers want and what every editorial style guide has recommended for decades. Self-contained sections that do not depend on distant context help anyone who lands mid-page from a search result or skims rather than reads linearly. Semantic HTML — real headings, real lists, real buttons and labels — is the same foundation that makes a page work with screen readers and keyboard navigation, so machine-readability and accessibility are substantially the same project. Honest structured data forces clarity about what the page actually is. The apparent tensions are worth naming and are mostly resolvable. Visual richness and interactivity are not prohibited: the pattern is to render the substance server-side and layer enhancement on top, so the content exists for parsers while the experience delights people. Persuasive narrative and answer-first structure can coexist by leading each section with the claim and using the supporting prose to build the case — you lose the mystery-novel structure, which was rarely serving readers anyway. The one genuine trade-off is that some designs which hide information behind interactions (accordions that load content on click, data only rendered after a chart interaction) need rethinking so the information also exists in the markup — which, again, is what accessibility guidance already asked for. The honest summary: designing for machines is mostly a forcing function for clarity, and teams that do it well report their human metrics improving too.

Q: What is the first thing to fix if my site is invisible to AI agents?

Server-render the content that matters, then check what a parser actually sees — in that order, because everything else is wasted if the substance is not in the HTML. The single highest-return diagnostic takes minutes: fetch your important pages the way a simple parser does (curl the URL, or view source rather than the rendered DOM) and read what comes back. If your headline, your key facts, and your body content are present, you have a foundation to build on. If you get an empty shell that populates only after JavaScript runs, that is your problem and nothing else matters until it is fixed, because a meaningful share of machine visitors will never execute your scripts and will simply see nothing. Fix it by moving the substantive content to server-side rendering — via server components, static generation, or a server-rendered architecture — keeping client-side code for genuine interactivity layered on top. Once the content is actually there, the next three fixes in order: make the structure semantic (real headings in a sensible hierarchy, real lists, meaningful element choices rather than a sea of divs) so a parser can understand the shape of the page; restructure the writing to be answer-first with self-contained sections so extraction produces correct, quotable claims; and add honest schema.org markup so machines know what the content is rather than inferring. After that come the entry points (sitemap, deliberate robots policy, llms.txt) and, if relevant, the agent-facing API. One more early step worth doing in parallel because it changes how you evaluate everything else: segment bot traffic from human traffic in your analytics. Most teams have never done this, and until you can see the two populations separately you cannot tell whether any of your changes are working — you are optimising a number that mixes an audience you are trying to serve with one you are trying to filter.