When an AI assistant cites your documentation, four things had to succeed first: discover, permit, fetch, parse. Each fails silently, and nothing in that chain is a ranking lever — it is retrieval plumbing. Baseline from our own audit: only 3 of our 11 domains serve a real llms.txt.

What GEO Actually Is (and What It Isn’t)

Generative-engine optimization has a research definition. The GEO paper (arXiv:2311.09735), accepted at KDD 2024, frames GEO as a black-box optimization framework, ships the GEO-bench evaluation set, and reports visibility gains “up to 40%” — while noting that “the efficacy of these strategies varies across domains.” That is a measurement framing, not a promise: the paper does not guarantee citations or rankings to any specific site.

For engineers, GEO reduces to retrievability engineering. Google’s AI features documentation is blunt: to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Search with a snippet — “There are no additional technical requirements.” There is no secret lever. The title’s promise — “gets your docs cited” — should be read strictly as citable: these are necessary conditions, not sufficient ones.

How an Agent Reads Your Site: Four Stages, Four Silent Failures

A citation is only possible when four stages succeed in order:

  1. Discover. Something points at your URLs: links, sitemaps, an llms.txt index. Failure: nothing points anywhere, and the fetch never happens.
  2. Permit. The crawler’s policy engine matches its product token against your robots.txt. Failure: your rules sit in a group the token never matches.
  3. Fetch. A plain HTTP GET. Failure: HTTP 200 with the wrong body — a soft-404. No error surfaces on either side.
  4. Parse. An HTML page wraps its content in navigation, ads and script, and converting it back to clean text is, in the proposal’s words, “difficult and imprecise” — which is why it asks for a clean markdown version of each page. Failure: boilerplate buries the content.

Every failure here is silent: the server logs a normal request, the agent logs a normal fetch, and the citation simply never becomes possible. Design for the strictest client — a plain GET with no JavaScript — because relying on rendering is betting on undocumented behavior. Stages 1 and 4 get artifacts below; stage 2 gets two sections.

llms.txt at v2: The Anatomy That Matters

The llms.txt v2 spec places the file at /llms.txt — or at any subpath such as /docs/llms.txt, covering the URLs under its path; when more than one applies, agents should use the most specific one. Required order: an optional byte-order mark; one H1 with the project name (“This is the only required section”); a blockquote summary; zero or more free markdown sections; zero or more H2-delimited file-list sections whose items are [name](url) optionally followed by : notes. An ## Optional section is the convention for secondary links. The size rule is the design: “The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed.”

Adoption is documented on the spec page: “thousands of sites publish an llms.txt file,” platforms such as Mintlify generate one automatically, and Chrome’s Lighthouse audits sites for one as part of its agentic browsing checks. OpenAI, Anthropic, and Gemini publish llms.txt for their own developer docs, and Cloudflare’s docs serve an llms.txt index with per-page markdown mirrors.

Skeleton — the BOM is optional and omitted here:

# Example Product Docs

> Documentation for the Example Product API: authentication, webhooks, and the REST reference.

## Guides

- [Quickstart](/docs/quickstart): first authenticated request in five minutes
- [Rate limits](/docs/rate-limits): quotas, headers, and 429 handling

## Optional

- [Changelog](/changelog): release notes, newest first
- [Status page](/status): uptime history and incidents

One H1, a summary, and links with notes — nothing else is required.

Markdown Mirrors: The .md Convention and How Agents Discover It

The same spec defines a clean markdown version served “at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md).” URLs without a file name use index.html.md or index.md. All four forms count.

Discovery is explicit, via two link relations — rel="alternate" with type="text/markdown" for the mirror and rel="describedby" for the llms.txt covering the page — as HTML <link> elements:

<link rel="alternate" type="text/markdown" href="https://example.com/docs/page.html.md">
<link rel="describedby" href="https://example.com/docs/llms.txt">

or as an HTTP header, verbatim from the spec:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The point is parse determinism: extracting content from HTML means stripping boilerplate, nav, and templates. A mirror that answers with a markdown content type removes that guesswork. In our audit, exactly 1 of 11 domains answered 200 on a mirror path — and it is the same domain with a soft-404 llms.txt, which is why verification dominates the rest of this guide.

The 2026 Crawler Roster: Eleven Rows, Four Operators

Every major operator runs multiple crawlers with distinct jobs and independent controls, so “one robots.txt line per vendor” is wrong at least three times over. Table A is the roster; the subsections cite each operator’s own pages.

Product token Operator Documented purpose robots.txt effective control? Real control surface
OAI-SearchBot OpenAI Surfaces sites in ChatGPT search answers; opted-out sites can still appear as navigational links Yes robots.txt group; per-bot IP JSON endpoint
GPTBot OpenAI Crawls content that may be used in training generative AI foundation models Yes robots.txt group; per-bot IP JSON endpoint
ChatGPT-User OpenAI User-initiated visits, incl. Custom GPTs; not automatic crawling No — “robots.txt rules may not apply” None documented for site owners; IP endpoint for verification
OAI-AdsBot OpenAI Validates landing pages submitted as ads; “only visits pages submitted as ads”, and its data is not used for training Ads-scoped only Not a crawling policy question; published IP endpoint
ClaudeBot Anthropic Training corpus; restricting signals exclusion from future training datasets Yes — explicit ClaudeBot rules override the general training-preference signal robots.txt; machine-readable training-preference signal; published IP list
Claude-User Anthropic User-initiated retrieval; disabling prevents retrieval for user queries Yes — documented as a blockable crawler robots.txt; published IP list
Claude-SearchBot Anthropic Indexes content for search optimization Yes robots.txt; published IP list
PerplexityBot Perplexity Crawls for search results Yes robots.txt; published IP ranges; edge/WAF rules
Perplexity-User Perplexity User-initiated fetches No — “generally ignores robots.txt rules” Published IP ranges; edge/WAF rules
Googlebot Google Crawls and indexes for Google Search; the eligibility baseline for AI-feature supporting links Yes robots.txt crawler rules; Search technical requirements
Google-Extended Google Control token for AI training/grounding in other Google systems Yes — a robots.txt token, not a separate crawler Token applied over Google’s existing user agents

Two rows — ChatGPT-User and Perplexity-User — are user-initiated fetchers for which robots.txt is explicitly not the control; one row, OAI-AdsBot, only fetches pages submitted as ads; the rest are configurable in robots.txt, and Google-Extended is a token rather than a crawler.

OpenAI: OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot

OpenAI’s crawler documentation documents four bots with independent robots.txt settings. OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features” — opted-out sites “will not be shown in ChatGPT search answers, though can still appear as navigational links.” GPTBot crawls content that may be used in training foundation models. ChatGPT-User handles user-initiated visits and “is not used for crawling the web in an automatic fashion.” OAI-AdsBot is ads-scoped: it “only visits pages submitted as ads” and “the data collected by OAI-AdsBot is not used to train generative AI foundation models”, so it does not appear in the robots.txt skeleton below. For search, “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” Each bot publishes an IP JSON endpoint, linked from the same page.

Anthropic: ClaudeBot, Claude-User, Claude-SearchBot

Anthropic’s crawler support article documents three robots. ClaudeBot builds the training corpus — restricting it “signals that the site’s future materials should be excluded from our AI model training datasets.” Claude-User performs user-initiated retrieval; disabling it “may reduce your site’s visibility for user-directed web search.” Claude-SearchBot indexes for search optimization. Anthropic supports the non-standard Crawl-delay extension, and — as of October 30, 2026 — recognizes a machine-readable training-preference signal following an IETF aipref draft, with explicit ClaudeBot rules taking precedence. Crawler IPs are published as JSON.

Perplexity: PerplexityBot and Perplexity-User

Perplexity’s bot guide runs two bots. PerplexityBot crawls for search results; Perplexity-User handles user-initiated fetches and “generally ignores robots.txt rules” — robots.txt is not its control path. Perplexity publishes IP ranges for both, and the guide demonstrates an edge/WAF rule matching the user agent against those ranges.

Google: Googlebot, Google-Extended, and AI Overviews

Per Google’s AI features documentation: “To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet… There are no additional technical requirements.” Both features “may use a ‘query fan-out’ technique — issuing multiple related searches across subtopics and data sources,” so related pages feed answers too. Best practice is unchanged — crawlability, textual content, structured data matching visible text — and AI-feature appearances count in overall Search Console traffic. Separately, Google-Extended controls AI training/grounding in other Google systems; per Google’s crawler documentation it “doesn’t have a separate HTTP request user agent string” — crawling uses existing Google user agents and the token acts as the control.

robots.txt Mechanics: What RFC 9309 Actually Says

Interop is specified by RFC 9309. Four rules do most of the work:

  • Group merging: when a token matches multiple groups, they “MUST be combined into one group” — later blocks merge with earlier ones; they do not override.
  • Case-insensitive matching: crawlers “MUST use case-insensitive matching to find the group that matches the product token,” and the token “SHOULD be a substring in the User-Agent header” — so gptbot matches GPTBot.
  • Most-octets path matching: for paths, “The most specific match found MUST be used,” measured in octets — /docs/private beats /docs.
  • Fallback: with no matching group, the * group applies.

Consequence: a *-only file gives every token one policy, and a typo’d token silently falls through to *. Operationally: OpenAI’s ~24-hour adjust window applies to search; Anthropic honors Crawl-delay despite its non-standard status; explicit ClaudeBot rules override the general training-preference signal. Starter skeleton — all nine AI tokens named plus the * fallback. Consecutive User-agent lines form one group, so every token here gets the same explicit rules; split into per-token groups when policies diverge. The rules are placeholders, not a recommendation:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Where robots.txt Stops Working: User-Initiated Fetches

Two of the nine tokens are user-initiated fetchers, and for both, robots.txt is explicitly not the control: OpenAI documents that ChatGPT-User acts on user request, where “robots.txt rules may not apply,” and Perplexity states that Perplexity-User “generally ignores robots.txt rules.” robots.txt is policy for automatic index-and-build crawlers; it is not a boundary for a user asking a question.

The documented controls at that layer are IP-based. Perplexity publishes ranges for Perplexity-User and PerplexityBot; Anthropic and OpenAI publish per-bot IP lists too. The same Perplexity guide carries one operational caveat: Cloudflare’s Bot Fight Mode “can’t be skipped with WAF custom rules,” so an allowlist admitting a fetcher won’t override it. For packaging, Cloudflare AI Crawl Control (docs last updated Aug 14, 2026) monitors AI crawler traffic, sets per-crawler allow/block rules, tracks robots.txt compliance, and offers pay-per-crawl in private beta — working automatically on all Cloudflare plans. Which crawlers to admit is a policy decision this guide does not make for you.

The Audit: 11 Domains, Plain GETs, and What Actually Failed

Retrieval readiness is measurable with plain HTTP, so we measured our own estate. At 2026-11-17T12:04:22Z we ran one GET per path — /llms.txt, /robots.txt, /index.md, /index.html.md — with UA NiteAgentAudit/1.0, no JavaScript, no cookies, recording status, Content-Type and response size for every request, plus the first line of each body.

Check Result (11 domains) Notable detail
HTTPS reachable 11/11 All domains answered over HTTPS
/llms.txt HTTP 200 4/11 niteagent.com, smarthomefieldguide.com, nocodeinsider.com, quantbrainai.net
Real text/plain llms.txt 3/11 niteagent.com: 1,165 B starting # NiteAgent; smarthomefieldguide.com: 916 B; nocodeinsider.com: 291 B
/llms.txt 200 but not markdown 1/11 quantbrainai.net: text/html — the site’s 99,127-byte HTML homepage
/robots.txt HTTP 200 9/11 404s on codeintel.dev and healthywallet.com
robots.txt names ≥1 AI token 1/11 niteagent.com only: GPTBot, ClaudeBot, PerplexityBot, each Allow: /
/index.md or /index.html.md 200 1/11 quantbrainai.net, on both forms

The 200-versus-real-file gap is the story: four domains answered, three had a usable index.

The soft-404 deserves its own callout. quantbrainai.net’s /llms.txt returned HTTP 200 with a body that is the site’s HTML homepage: the origin logs success, the agent receives roughly 99 KB of markup where it expected a small index, and no error surfaces anywhere. Status codes alone cannot distinguish a working llms.txt from an SPA catch-all route — you need status, Content-Type, and body size together.

What to Deploy: A Verifiable Retrieval Checklist

  1. robots.txt with explicit token groups. Use the §6 skeleton: all nine AI tokens named, plus *. Check: fetch /robots.txt and confirm each token appears.
  2. A small root llms.txt. Use the §3 skeleton; keep detail behind links so the file stays within a context budget. For deep docs trees, add subpath files (/docs/llms.txt) — most specific wins.
  3. Markdown mirrors with discovery. Serve page.md / page.html.md (index.md for extensionless URLs) and advertise them with the <link> elements or Link: header from §4.
  4. Verify with plain GETs — status, Content-Type, size. This is the check that catches soft-404s; the quantbrainai.net case passes any status-only test.
  5. Set per-vendor posture deliberately. Search, training, and user-initiated fetch are separate controls per operator — OpenAI’s three policy crawlers (plus the ads-scoped OAI-AdsBot), Anthropic’s three plus the training-preference signal, Google’s Google-Extended. For user-initiated fetchers, the control is IP/WAF, not robots.txt. The posture choice is yours.
  6. Monitor. Cloudflare AI Crawl Control gives per-crawler traffic and robots.txt-compliance tracking on all plans; access logs work too — watch statuses and content types on /llms.txt and /index.md.

Run this against your own domain:

# One line per path: status, content type, body size.
for path in llms.txt robots.txt index.md; do
  curl -sS -o /tmp/geo_body -A AuditCheck/1.0 \
    -w "/${path} -> HTTP %{http_code} | %{content_type} | %{size_download} B\n" \
    "https://your-domain.example/${path}"
done
head -c 160 /tmp/geo_body; echo

A 200 with text/html and a five-figure byte count on /llms.txt is the quantbrainai.net pattern — fix the route before anything else.

FAQ

Does llms.txt improve ranking? No ranking effect is claimed anywhere in the proposal — it defines an index format, nothing more. Google’s position matches: AI Overviews/AI Mode eligibility requires only indexing and snippet eligibility, with “no additional technical requirements.”

Does blocking GPTBot remove me from ChatGPT search? No. GPTBot is the training crawler; OAI-SearchBot is the search crawler, with independent robots.txt settings. Search presence follows OAI-SearchBot’s group; training participation follows GPTBot’s.

Can robots.txt stop user-initiated fetches? Not reliably. OpenAI documents that ChatGPT-User requests mean “robots.txt rules may not apply,” and Perplexity states Perplexity-User “generally ignores robots.txt rules.” The effective controls there are published IP ranges and edge/WAF rules.

One llms.txt at the root, or one per docs section? Either. The file can live at any subpath and covers the URLs under that path; when multiple files apply, agents use the most specific one.

How fast do robots.txt changes take effect? The only documented timing here is OpenAI’s: “~24 hours” for its search systems to adjust after a robots.txt update. Other operators’ propagation windows are not documented on the pages cited.

What does Google need for AI Overviews and AI Mode? Indexing and snippet eligibility in Search — “no additional technical requirements.” Because of query fan-out, related pages across subtopics can also feed an answer, so sitewide crawlability matters, not just the landing page.

Is Crawl-delay part of the robots.txt standard? No — it is a non-standard extension. Anthropic documents support for it; RFC 9309 does not define it.

The Bottom Line

You don’t optimize your way into citations — you unblock them: a citation requires discover → permit → fetch → parse to succeed, and no directive, file, or setting in this guide is a ranking lever. Deployment priority:

  1. Publish robots.txt with explicit groups for all nine AI tokens plus *.
  2. Publish a small llms.txt at the root — subpath files for deep trees.
  3. Serve markdown mirrors and advertise them with the <link>/Link: discovery headers.
  4. Curl-verify every path — status, Content-Type, size — and kill soft-404s.
  5. Choose per-vendor posture deliberately — search, training, and user-initiated fetch are separate controls — and monitor what actually fetches.

The measurable news is that most sites, including most of ours, haven’t done the basics: 4 of 11 domains returned HTTP 200 for llms.txt, only 3 were real. The basics are what currently separates a retrievable site from an unretrievable one.

How This Guide Was Built

The audit numbers come from the author’s own checks: plain HTTP GETs, one request per URL, no JavaScript, no cookies, UA NiteAgentAudit/1.0, run at 2026-11-17T12:04:22Z across our 11 domains. No AI-crawler traffic data or server logs were available for any domain, so this article makes no claims about how agents actually behaved on these sites — only whether the retrieval plumbing answers. Every spec and operator claim was verified on 2026-11-17 against the operator pages linked inline, and quotes are verbatim from those sources. The GEO research results belong to their authors; nothing here re-ran their benchmarks.

← Back to all posts