ChatGPT
OpenAI · ChatGPT search
OAI-SearchBotSearch: finds and cites pagesAllowedChatGPT-UserFetches a page a person asks aboutAllowedGPTBotTrainingAllowed
By Oliver Thompson at Analogue Media ·
To a model, that sentence is 9 numbers: 32 3438 6884 395 290 14656 484 6052 13
If ChatGPT, Claude and Google’s AI search were a website’s only readers, what would it be? This is one answer: answer engine optimisation (AEO), built for real and open to every one of them.
Move across the headline to read it the way a language model does.Tap the headline to read it the way a language model does.
The three readers
Each sends its own crawlers: one to search, one to fetch a page a person asks about, one to learn. This site’s robots.txt is checked below, live, for every one of them.
OpenAI · ChatGPT search
OAI-SearchBotSearch: finds and cites pagesAllowedChatGPT-UserFetches a page a person asks aboutAllowedGPTBotTrainingAllowedAnthropic · Claude’s web search
Claude-SearchBotSearch: indexes pages for answersAllowedClaude-UserFetches a page a person asks aboutAllowedClaudeBotTrainingAllowedGoogle · AI Overviews, AI Mode and Gemini
GooglebotSearch, AI Overviews and AI ModeAllowedGoogle-ExtendedGemini training and groundingAllowed[Figure: a map of the 63 passages on this website, placed by meaning. It is for people; a machine needs only the words.]
Retrieval, in your browser
Every passage on this site sits on this map by meaning, the way an answer engine stores it. Ask a question and watch it land among the passages that answer it, then see which one would be quoted.
Fourteen answers
Each answer opens with the passage an answer engine would quote, and every paragraph has its own address, so it can be cited on its own. Hover a paragraph for its number.
¶1.1Answer engine optimisation (AEO) is the practice of making a website easy for AI answer engines to find, understand and quote. The answer engines most people use are ChatGPT, Claude, and Google’s AI Overviews and AI Mode, alongside Gemini, Perplexity and Microsoft Copilot. Where search engine optimisation aims for a place in a list of links, AEO aims for a place inside the answer, as a cited source.
¶1.2An answer engine does not hand a person ten blue links. It reads several pages on their behalf, writes one answer and names a handful of sources beside it. The page that is quoted gets the credit, and often the visit; the pages that were only read get nothing.
¶1.3Answer engine optimisation is also called generative engine optimisation (GEO), AI search optimisation and LLM optimisation. The names differ but the job is the same: to be the clearest and most trustworthy source on the questions your customers ask.
¶2.1AEO is built on SEO rather than replacing it. Google says a page must be indexed and eligible to be shown with a snippet to appear in AI Overviews or AI Mode, and the other answer engines draw on search indexes of their own. A page that search engines cannot crawl, index or trust will not be quoted by an answer engine either.
¶2.2The difference between AEO and SEO is in what wins. SEO rewards the page that best matches a search; AEO rewards the passage that best answers a question. An answer engine often lifts a single paragraph, a definition, a figure or a step, so every paragraph on a page has to make sense on its own.
¶2.3SEO is measured in rankings and clicks. AEO is measured in citations: how often a page is named as a source inside an answer. Microsoft began reporting citations in Copilot and Bing’s AI answers to website owners in February 2026, through Bing Webmaster Tools.
¶3.1ChatGPT, Claude and Google’s AI search each find websites through a search index and a crawler of their own. ChatGPT search uses OpenAI’s OAI-SearchBot, Claude uses Anthropic’s Claude-SearchBot, and Google’s AI Overviews and AI Mode use Googlebot and the Google Search index. A website that lets all three crawlers in, and is indexed by them, can be found and quoted by all three.
¶3.2ChatGPT finds pages for its search answers with OAI-SearchBot. OpenAI says a site that blocks OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link. When a person asks ChatGPT to open a page, ChatGPT-User fetches it for them.
¶3.3Claude searches the web with the help of Claude-SearchBot, which indexes pages to improve Claude’s search results, and Claude-User, which fetches a page when a person asks Claude about it. Anthropic says its crawlers honour robots.txt, and that blocking Claude-SearchBot may reduce a site’s visibility in Claude’s answers.
¶3.4Google’s AI Overviews and AI Mode are built on Google Search. Google says a page needs only to be indexed and eligible to be shown with a snippet; there are no extra requirements. AI Mode leans on query fan-out, running many related searches at once, so a page can be cited for a part of the question nobody typed.
¶3.5Gemini, Google’s assistant, is governed separately. Google-Extended, a name used only in robots.txt, controls whether a site is used for training and grounding in Gemini and some other Google systems. Blocking Google-Extended does not remove a site from Google Search, AI Overviews or AI Mode.
OAI-SearchBot Search: finds and cites pagesChatGPT-User Fetches a page a person asks aboutGPTBot TrainingClaude-SearchBot Search: indexes pages for answersClaude-User Fetches a page a person asks aboutClaudeBot TrainingGooglebot Search, AI Overviews and AI ModeGoogle-Extended Gemini training and grounding¶4.1AI Overviews and AI Mode are both part of Google Search. AI Overviews is a short AI-written summary shown above some ordinary search results, only when Google’s systems judge that it adds something. AI Mode is a separate, conversational search in which a person asks follow-up questions and gets a fuller answer. Both draw on the Google Search index and link to their sources.
¶4.2Google says both AI Overviews and AI Mode may use query fan-out, running several related searches across the parts of a question to build a response. That lets them show a wider and more varied set of links than a classic page of results, so a page can be linked for one part of a question even when it would not rank for the whole.
¶4.3The requirements are the same for both: a page must be indexed and eligible to be shown in Google Search with a snippet. Both count towards a site’s figures in Google Search Console, and both follow the same controls: nosnippet, data-nosnippet, max-snippet and noindex.
¶5.1Answer engines choose what to quote by retrieval. Before writing, an answer engine searches an index for passages that match the question, ranks them, and hands the best few to a language model as its sources. The model writes an answer grounded in those passages and cites them. A page is quoted when one of its passages ranks among the best and states the answer plainly.
¶5.2Google describes a technique called query fan-out, in which the model turns one question into several related searches and gathers results for each. A page can be chosen for a part of the question the person never typed, which rewards websites that cover a subject thoroughly rather than one phrase.
¶5.3Retrieval matches by meaning as well as by words. Questions and passages can be turned into vectors, lists of numbers that place text with similar meaning close together, so a page about being quoted by ChatGPT can match a question about appearing in AI answers. The search on this page works the same way, in miniature, in your browser.
¶6.1A machine reads the HTML a server sends: the text, the headings, the links, the alt text of images and any structured data. Some AI crawlers do not run JavaScript, so words that appear only after scripts run may never be seen by them. Colour, layout and animation are, for the most part, invisible to a machine.
¶6.2A 2024 study by Vercel and MERJ found that the crawlers OpenAI and Anthropic then used fetched JavaScript files but did not run them, while Google’s crawler did. The safe rule is that every word that matters should be in the HTML of the first response, before any script runs.
¶6.3Headings tell a machine what each part of a page is about, so a heading written as the question it answers is the most useful kind. Lists, tables and definitions are easy to lift whole. A picture is understood only through its alt text and its caption.
¶6.4Every word on this website is in the HTML when it arrives. Scripts add the demonstrations and the machine view, but with scripts turned off every answer, table and link is still there. The machine view, at the top of the page, shows roughly what a crawler keeps.
¶7.1To be quoted, put the answer first. Start each section with one or two sentences that answer its question completely, in plain words, then give the detail. Answer engines lift passages out of their page, so each paragraph should name its subject rather than lean on “it” or “this”, and keep to one idea.
¶7.2Specific beats general. A price, a date, a measurement, a place or a named source gives an answer engine something to quote and a reason to trust it. “Delivered in three to five working days across mainland Britain” can be quoted; “fast delivery” cannot.
¶7.3Say who is speaking and when. A visible author, a date of publication and a date of the last change let a reader, human or machine, judge whose word a page is and whether it is still current.
¶7.4Be the source. Answer engines cite pages that add something: original figures, first-hand experience, a clear definition, a worked example, a price. A page that only restates what others have written gives an answer engine no reason to name it.
¶7.5None of this is a trick for machines. Google says there is no need to break content into tiny pieces or to rewrite it for AI. Clear, specific, well organised writing serves both readers at once, which is why it works.
¶8.1Structured data helps machines be sure of the facts, but it is not a ticket into AI answers. Google says no special schema.org markup is needed for AI Overviews or AI Mode, and that structured data must match the visible text. It still earns rich results in search and states plainly who you are, what you offer, where and when.
¶8.2Structured data is written in JSON-LD, using the vocabulary of schema.org: a business is an Organization with an address and opening hours, a product has a price and availability, a question has an accepted answer. The value is in being exact, so that a machine never has to guess.
¶8.3This page carries one connected graph of structured data: the website, the studio behind it, this page, each question and its answer, and each term it defines. Every value in the graph is also written on the page, where people can read it.
¶9.1llms.txt is a proposed Markdown file, placed at the root of a website, that gives language models a short guide to the site: its name, a summary and links to its most useful pages. Jeremy Howard of Answer.AI proposed it on 3 September 2024. No major answer engine has said it uses llms.txt to find or rank pages.
¶9.2Google says its Search does not use llms.txt files, and that making one will neither help nor harm a website. The format’s real use today is in developer tools, where a coding assistant can load a project’s documentation from it in one go.
¶9.3The llms.txt proposal also suggests a clean Markdown copy of each page, at the same address with .md added. This website has both, at /llms.txt and /llms-full.txt, because they cost nothing and show what a site is to a reader that wants only the words.
¶10.1Each AI company sends separate crawlers for separate jobs: one to build the search index its answers draw on, one to gather pages for training, and one to fetch a page when a person asks about it. A website can welcome the search crawlers, which lead to citations, and decide about training separately.
¶10.2OpenAI uses OAI-SearchBot for ChatGPT search, GPTBot for training and ChatGPT-User when a person’s request needs a page. Anthropic uses Claude-SearchBot for search, ClaudeBot for training and Claude-User for requests. Perplexity uses PerplexityBot for search and Perplexity-User for requests.
¶10.3Google’s AI Overviews and AI Mode use Googlebot, the same crawler as Search, so blocking Googlebot removes a page from both. Google-Extended is not a crawler but a name in robots.txt that controls use of a site in training and grounding for Gemini and some other Google systems, without affecting Search.
¶10.4Fetchers that act for a person may not follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
| Company | Crawler | What it does | Follows robots.txt | Here |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Finds pages to show and cite in ChatGPT search | Yes | Welcome |
| OpenAI | GPTBot | Gathers pages that may be used to train OpenAI’s models | Yes | Welcome |
| OpenAI | ChatGPT-User | Fetches a page when a person’s request in ChatGPT needs it | May not apply | Welcome |
| Anthropic | Claude-SearchBot | Indexes pages to improve Claude’s search results | Yes | Welcome |
| Anthropic | ClaudeBot | Gathers pages that may be used to train Claude | Yes | Welcome |
| Anthropic | Claude-User | Fetches a page when a person asks Claude about it | Yes | Welcome |
| Perplexity | PerplexityBot | Finds pages to show and link in Perplexity’s answers | Yes | Welcome |
| Perplexity | Perplexity-User | Fetches a page to answer a person’s question | Generally not | Welcome |
Googlebot | Crawls for Search, AI Overviews and AI Mode | Yes | Welcome | |
Google-Extended | A robots.txt name, not a crawler: use in Gemini training and grounding | Yes | Welcome | |
| Microsoft | Bingbot | Crawls for Bing, whose index grounds Copilot’s answers | Yes | Welcome |
| Apple | Applebot | Crawls for Siri, Spotlight and Safari suggestions | Yes | Welcome |
| Apple | Applebot-Extended | A robots.txt name, not a crawler: use in training Apple’s models | Yes | Welcome |
| Common Crawl | CCBot | Builds an open archive of the web that many models learn from | Yes | Welcome |
¶11.1Block the search crawlers and a website drops out of AI answers; block the training crawlers and it does not. OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, while blocking GPTBot only signals that pages should not be used for training. A business that wants to be cited should let the search crawlers in, whatever it decides about training.
¶11.2The same split holds for Claude and Google. Claude-SearchBot serves Claude’s search and ClaudeBot gathers pages for training. Googlebot serves Google Search, AI Overviews and AI Mode, while Google-Extended controls use in training and grounding for Gemini and has no effect on Search.
¶11.3Blocking Googlebot to stay out of AI Overviews would remove a website from Google Search altogether. Google’s finer tools are its snippet controls: nosnippet and data-nosnippet keep chosen words out of AI answers and ordinary snippets alike.
¶11.4This website blocks nothing. Its robots.txt names every AI crawler and allows each one, and its Content-Signal line says yes to search, to use in answers and to training.
¶12.1Google lists several things that are not needed to appear in its AI features: llms.txt files, special AI markup, special schema.org types, breaking content into tiny chunks and rewriting content for AI. Arranging inauthentic mentions of a brand is called out too. What decides is a page that can be crawled, indexed and shown with a snippet, and that answers well.
¶12.2Being shown with a snippet is a real lever. A robots meta tag of max-snippet:-1 lets search engines quote as much of a page as they need, while nosnippet, data-nosnippet and a short max-snippet limit what can be quoted, in AI answers as well as in ordinary results.
¶13.1To measure visibility in AI answers, look for answer engines in your visitor numbers and for your name in their answers. Visits from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com show as referrals in most analytics tools. Bing Webmaster Tools counts citations in Copilot, and Google Search Console includes AI Overviews and AI Mode in its search totals.
¶13.2Then ask. Put the questions your customers ask into ChatGPT, Claude, Perplexity, Gemini and Copilot, and note who is cited. Answers change from one asking to the next, so repeat each question and count over weeks rather than trusting a single result.
¶14.1A website designed for AI looks, underneath, like a very well made website for people: every answer in the HTML, a heading for each question, the answer first, plain facts with dates and sources, quick pages and nothing hidden behind a script. What it adds is a Markdown twin of each page, one connected graph of structured data and an open door for the crawlers that lead to citations.
¶14.2This website is built that way. Every word is in the HTML when it arrives, each page has a Markdown twin, the structured data describes the page as one graph, robots.txt welcomes each AI crawler by name and the sitemap dates every change. Then, because people still choose who to buy from, it is made to be a pleasure to read.
Vocabulary
Under each word, the tokens a model reads instead.
Anatomy
The page you are reading is one of seven ways this website speaks to machines. Open each to see it exactly as a crawler does.
curl -H "Accept: text/markdown" https://aeo.observer/ returns this page as Markdown.
¶A.1aeo.observer is an idea made by Analogue Media, a web design studio in Bath, England: a website designed first for the machines that answer questions, to see what such a website would be. It was written by Oliver Thompson, published on 22 November 2026, and every fact on it was checked against its source that week.
¶A.2The name is the job: aeo.observer watches how answer engines read the web, and is built to be read by them. To a language model the name is four tokens, “ae”, “o”, “.ob” and “server”, which is fitting for a website that serves machines first.
¶A.3The site is its own demonstration. The words are in the HTML when the page arrives, every page has a Markdown twin, the structured data describes the page as one connected graph, and the search on this page runs entirely in your browser, with nothing sent anywhere.