aeo.observer

By Oliver Thompson at Analogue Media ·

A website designed for the machines that answer.

To a model, that sentence is 9 numbers: 32 3438 6884 395 290 14656 484 6052 13

If ChatGPT, Claude and Google’s AI search were a website’s only readers, what would it be? This is one answer: answer engine optimisation (AEO), built for real and open to every one of them.

Move across the headline to read it the way a language model does.Tap the headline to read it the way a language model does.

Ask this website a question
What a crawler is sent
24.5 KB5,763 tokens0 scripts run

The three readers

Built first for ChatGPT, Claude and Google’s AI search.

Each sends its own crawlers: one to search, one to fetch a page a person asks about, one to learn. This site’s robots.txt is checked below, live, for every one of them.

ChatGPT

OpenAI · ChatGPT search

  • OAI-SearchBotSearch: finds and cites pagesAllowed
  • ChatGPT-UserFetches a page a person asks aboutAllowed
  • GPTBotTrainingAllowed

Claude

Anthropic · Claude’s web search

  • Claude-SearchBotSearch: indexes pages for answersAllowed
  • Claude-UserFetches a page a person asks aboutAllowed
  • ClaudeBotTrainingAllowed

Google AI

Google · AI Overviews, AI Mode and Gemini

  • GooglebotSearch, AI Overviews and AI ModeAllowed
  • Google-ExtendedGemini training and groundingAllowed

[Figure: a map of the 63 passages on this website, placed by meaning. It is for people; a machine needs only the words.]

Retrieval, in your browser

Ask this website.

Every passage on this site sits on this map by meaning, the way an answer engine stores it. Ask a question and watch it land among the passages that answer it, then see which one would be quoted.

Fourteen answers

Written to be lifted.

Each answer opens with the passage an answer engine would quote, and every paragraph has its own address, so it can be cited on its own. Hover a paragraph for its number.

  1. 01What is answer engine optimisation (AEO)?
  2. 02Is AEO different from SEO?
  3. 03How do ChatGPT, Claude and Google’s AI search find a website?
  4. 04What is the difference between AI Overviews and AI Mode?
  5. 05How do answer engines choose what to quote?
  6. 06What does a machine read on a web page?
  7. 07How should a page be written to be quoted?
  8. 08Does structured data help with AI answers?
  9. 09What is llms.txt, and does it work?
  10. 10Which AI crawlers visit websites?
  11. 11Should you block AI crawlers?
  12. 12What don’t you need to do?
  13. 13How do you measure visibility in AI answers?
  14. 14What does a website designed for AI look like?

What is answer engine optimisation (AEO)?

Checked · Its own page

¶1.1Answer engine optimisation (AEO) is the practice of making a website easy for AI answer engines to find, understand and quote. The answer engines most people use are ChatGPT, Claude, and Google’s AI Overviews and AI Mode, alongside Gemini, Perplexity and Microsoft Copilot. Where search engine optimisation aims for a place in a list of links, AEO aims for a place inside the answer, as a cited source.

¶1.2An answer engine does not hand a person ten blue links. It reads several pages on their behalf, writes one answer and names a handful of sources beside it. The page that is quoted gets the credit, and often the visit; the pages that were only read get nothing.

¶1.3Answer engine optimisation is also called generative engine optimisation (GEO), AI search optimisation and LLM optimisation. The names differ but the job is the same: to be the clearest and most trustworthy source on the questions your customers ask.

Is AEO different from SEO?

Checked · Its own page

¶2.1AEO is built on SEO rather than replacing it. Google says a page must be indexed and eligible to be shown with a snippet to appear in AI Overviews or AI Mode, and the other answer engines draw on search indexes of their own. A page that search engines cannot crawl, index or trust will not be quoted by an answer engine either.

¶2.2The difference between AEO and SEO is in what wins. SEO rewards the page that best matches a search; AEO rewards the passage that best answers a question. An answer engine often lifts a single paragraph, a definition, a figure or a step, so every paragraph on a page has to make sense on its own.

¶2.3SEO is measured in rankings and clicks. AEO is measured in citations: how often a page is named as a source inside an answer. Microsoft began reporting citations in Copilot and Bing’s AI answers to website owners in February 2026, through Bing Webmaster Tools.

How do ChatGPT, Claude and Google’s AI search find a website?

Checked · Its own page

¶3.1ChatGPT, Claude and Google’s AI search each find websites through a search index and a crawler of their own. ChatGPT search uses OpenAI’s OAI-SearchBot, Claude uses Anthropic’s Claude-SearchBot, and Google’s AI Overviews and AI Mode use Googlebot and the Google Search index. A website that lets all three crawlers in, and is indexed by them, can be found and quoted by all three.

¶3.2ChatGPT finds pages for its search answers with OAI-SearchBot. OpenAI says a site that blocks OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link. When a person asks ChatGPT to open a page, ChatGPT-User fetches it for them.

¶3.3Claude searches the web with the help of Claude-SearchBot, which indexes pages to improve Claude’s search results, and Claude-User, which fetches a page when a person asks Claude about it. Anthropic says its crawlers honour robots.txt, and that blocking Claude-SearchBot may reduce a site’s visibility in Claude’s answers.

¶3.4Google’s AI Overviews and AI Mode are built on Google Search. Google says a page needs only to be indexed and eligible to be shown with a snippet; there are no extra requirements. AI Mode leans on query fan-out, running many related searches at once, so a page can be cited for a part of the question nobody typed.

¶3.5Gemini, Google’s assistant, is governed separately. Google-Extended, a name used only in robots.txt, controls whether a site is used for training and grounding in Gemini and some other Google systems. Blocking Google-Extended does not remove a site from Google Search, AI Overviews or AI Mode.

  • ChatGPT ChatGPT search
    • OAI-SearchBot Search: finds and cites pages
    • ChatGPT-User Fetches a page a person asks about
    • GPTBot Training
  • Claude Claude’s web search
    • Claude-SearchBot Search: indexes pages for answers
    • Claude-User Fetches a page a person asks about
    • ClaudeBot Training
  • Google AI AI Overviews, AI Mode and Gemini
    • Googlebot Search, AI Overviews and AI Mode
    • Google-Extended Gemini training and grounding

What is the difference between AI Overviews and AI Mode?

Checked · Its own page

¶4.1AI Overviews and AI Mode are both part of Google Search. AI Overviews is a short AI-written summary shown above some ordinary search results, only when Google’s systems judge that it adds something. AI Mode is a separate, conversational search in which a person asks follow-up questions and gets a fuller answer. Both draw on the Google Search index and link to their sources.

¶4.2Google says both AI Overviews and AI Mode may use query fan-out, running several related searches across the parts of a question to build a response. That lets them show a wider and more varied set of links than a classic page of results, so a page can be linked for one part of a question even when it would not rank for the whole.

¶4.3The requirements are the same for both: a page must be indexed and eligible to be shown in Google Search with a snippet. Both count towards a site’s figures in Google Search Console, and both follow the same controls: nosnippet, data-nosnippet, max-snippet and noindex.

How do answer engines choose what to quote?

Checked · Its own page

¶5.1Answer engines choose what to quote by retrieval. Before writing, an answer engine searches an index for passages that match the question, ranks them, and hands the best few to a language model as its sources. The model writes an answer grounded in those passages and cites them. A page is quoted when one of its passages ranks among the best and states the answer plainly.

¶5.2Google describes a technique called query fan-out, in which the model turns one question into several related searches and gathers results for each. A page can be chosen for a part of the question the person never typed, which rewards websites that cover a subject thoroughly rather than one phrase.

¶5.3Retrieval matches by meaning as well as by words. Questions and passages can be turned into vectors, lists of numbers that place text with similar meaning close together, so a page about being quoted by ChatGPT can match a question about appearing in AI answers. The search on this page works the same way, in miniature, in your browser.

What does a machine read on a web page?

Checked · Its own page

¶6.1A machine reads the HTML a server sends: the text, the headings, the links, the alt text of images and any structured data. Some AI crawlers do not run JavaScript, so words that appear only after scripts run may never be seen by them. Colour, layout and animation are, for the most part, invisible to a machine.

¶6.2A 2024 study by Vercel and MERJ found that the crawlers OpenAI and Anthropic then used fetched JavaScript files but did not run them, while Google’s crawler did. The safe rule is that every word that matters should be in the HTML of the first response, before any script runs.

¶6.3Headings tell a machine what each part of a page is about, so a heading written as the question it answers is the most useful kind. Lists, tables and definitions are easy to lift whole. A picture is understood only through its alt text and its caption.

¶6.4Every word on this website is in the HTML when it arrives. Scripts add the demonstrations and the machine view, but with scripts turned off every answer, table and link is still there. The machine view, at the top of the page, shows roughly what a crawler keeps.

How should a page be written to be quoted?

Checked · Its own page

¶7.1To be quoted, put the answer first. Start each section with one or two sentences that answer its question completely, in plain words, then give the detail. Answer engines lift passages out of their page, so each paragraph should name its subject rather than lean on “it” or “this”, and keep to one idea.

¶7.2Specific beats general. A price, a date, a measurement, a place or a named source gives an answer engine something to quote and a reason to trust it. “Delivered in three to five working days across mainland Britain” can be quoted; “fast delivery” cannot.

¶7.3Say who is speaking and when. A visible author, a date of publication and a date of the last change let a reader, human or machine, judge whose word a page is and whether it is still current.

¶7.4Be the source. Answer engines cite pages that add something: original figures, first-hand experience, a clear definition, a worked example, a price. A page that only restates what others have written gives an answer engine no reason to name it.

¶7.5None of this is a trick for machines. Google says there is no need to break content into tiny pieces or to rewrite it for AI. Clear, specific, well organised writing serves both readers at once, which is why it works.

Does structured data help with AI answers?

Checked · Its own page

¶8.1Structured data helps machines be sure of the facts, but it is not a ticket into AI answers. Google says no special schema.org markup is needed for AI Overviews or AI Mode, and that structured data must match the visible text. It still earns rich results in search and states plainly who you are, what you offer, where and when.

¶8.2Structured data is written in JSON-LD, using the vocabulary of schema.org: a business is an Organization with an address and opening hours, a product has a price and availability, a question has an accepted answer. The value is in being exact, so that a machine never has to guess.

¶8.3This page carries one connected graph of structured data: the website, the studio behind it, this page, each question and its answer, and each term it defines. Every value in the graph is also written on the page, where people can read it.

What is llms.txt, and does it work?

Checked · Its own page

¶9.1llms.txt is a proposed Markdown file, placed at the root of a website, that gives language models a short guide to the site: its name, a summary and links to its most useful pages. Jeremy Howard of Answer.AI proposed it on 3 September 2024. No major answer engine has said it uses llms.txt to find or rank pages.

¶9.2Google says its Search does not use llms.txt files, and that making one will neither help nor harm a website. The format’s real use today is in developer tools, where a coding assistant can load a project’s documentation from it in one go.

¶9.3The llms.txt proposal also suggests a clean Markdown copy of each page, at the same address with .md added. This website has both, at /llms.txt and /llms-full.txt, because they cost nothing and show what a site is to a reader that wants only the words.

Which AI crawlers visit websites?

Checked · Its own page

¶10.1Each AI company sends separate crawlers for separate jobs: one to build the search index its answers draw on, one to gather pages for training, and one to fetch a page when a person asks about it. A website can welcome the search crawlers, which lead to citations, and decide about training separately.

¶10.2OpenAI uses OAI-SearchBot for ChatGPT search, GPTBot for training and ChatGPT-User when a person’s request needs a page. Anthropic uses Claude-SearchBot for search, ClaudeBot for training and Claude-User for requests. Perplexity uses PerplexityBot for search and Perplexity-User for requests.

¶10.3Google’s AI Overviews and AI Mode use Googlebot, the same crawler as Search, so blocking Googlebot removes a page from both. Google-Extended is not a crawler but a name in robots.txt that controls use of a site in training and grounding for Gemini and some other Google systems, without affecting Search.

¶10.4Fetchers that act for a person may not follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.

The crawlers, what each does, and whether it follows robots.txt
CompanyCrawlerWhat it doesFollows robots.txtHere
OpenAIOAI-SearchBotFinds pages to show and cite in ChatGPT searchYesWelcome
OpenAIGPTBotGathers pages that may be used to train OpenAI’s modelsYesWelcome
OpenAIChatGPT-UserFetches a page when a person’s request in ChatGPT needs itMay not applyWelcome
AnthropicClaude-SearchBotIndexes pages to improve Claude’s search resultsYesWelcome
AnthropicClaudeBotGathers pages that may be used to train ClaudeYesWelcome
AnthropicClaude-UserFetches a page when a person asks Claude about itYesWelcome
PerplexityPerplexityBotFinds pages to show and link in Perplexity’s answersYesWelcome
PerplexityPerplexity-UserFetches a page to answer a person’s questionGenerally notWelcome
GoogleGooglebotCrawls for Search, AI Overviews and AI ModeYesWelcome
GoogleGoogle-ExtendedA robots.txt name, not a crawler: use in Gemini training and groundingYesWelcome
MicrosoftBingbotCrawls for Bing, whose index grounds Copilot’s answersYesWelcome
AppleApplebotCrawls for Siri, Spotlight and Safari suggestionsYesWelcome
AppleApplebot-ExtendedA robots.txt name, not a crawler: use in training Apple’s modelsYesWelcome
Common CrawlCCBotBuilds an open archive of the web that many models learn fromYesWelcome

Should you block AI crawlers?

Checked · Its own page

¶11.1Block the search crawlers and a website drops out of AI answers; block the training crawlers and it does not. OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, while blocking GPTBot only signals that pages should not be used for training. A business that wants to be cited should let the search crawlers in, whatever it decides about training.

¶11.2The same split holds for Claude and Google. Claude-SearchBot serves Claude’s search and ClaudeBot gathers pages for training. Googlebot serves Google Search, AI Overviews and AI Mode, while Google-Extended controls use in training and grounding for Gemini and has no effect on Search.

¶11.3Blocking Googlebot to stay out of AI Overviews would remove a website from Google Search altogether. Google’s finer tools are its snippet controls: nosnippet and data-nosnippet keep chosen words out of AI answers and ordinary snippets alike.

¶11.4This website blocks nothing. Its robots.txt names every AI crawler and allows each one, and its Content-Signal line says yes to search, to use in answers and to training.

What don’t you need to do?

Checked · Its own page

¶12.1Google lists several things that are not needed to appear in its AI features: llms.txt files, special AI markup, special schema.org types, breaking content into tiny chunks and rewriting content for AI. Arranging inauthentic mentions of a brand is called out too. What decides is a page that can be crawled, indexed and shown with a snippet, and that answers well.

¶12.2Being shown with a snippet is a real lever. A robots meta tag of max-snippet:-1 lets search engines quote as much of a page as they need, while nosnippet, data-nosnippet and a short max-snippet limit what can be quoted, in AI answers as well as in ordinary results.

Not needed for Google’s AI features

  • An llms.txt file
  • Special AI markup or “AI text files”
  • Special schema.org types
  • Content broken into tiny chunks
  • Content rewritten for AI
  • Inauthentic mentions of your brand

What decides

  • Pages crawlers are allowed to fetch
  • Pages indexed by search engines
  • Pages that may be shown with a snippet
  • Structured data that matches the page
  • Clear answers with real facts and sources
  • Being the original source

How do you measure visibility in AI answers?

Checked · Its own page

¶13.1To measure visibility in AI answers, look for answer engines in your visitor numbers and for your name in their answers. Visits from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com show as referrals in most analytics tools. Bing Webmaster Tools counts citations in Copilot, and Google Search Console includes AI Overviews and AI Mode in its search totals.

¶13.2Then ask. Put the questions your customers ask into ChatGPT, Claude, Perplexity, Gemini and Copilot, and note who is cited. Answers change from one asking to the next, so repeat each question and count over weeks rather than trusting a single result.

What does a website designed for AI look like?

Checked · Its own page

¶14.1A website designed for AI looks, underneath, like a very well made website for people: every answer in the HTML, a heading for each question, the answer first, plain facts with dates and sources, quick pages and nothing hidden behind a script. What it adds is a Markdown twin of each page, one connected graph of structured data and an open door for the crawlers that lead to citations.

¶14.2This website is built that way. Every word is in the HTML when it arrives, each page has a Markdown twin, the structured data describes the page as one graph, robots.txt welcomes each AI crawler by name and the sitemap dates every change. Then, because people still choose who to buy from, it is made to be a pleasure to read.

Vocabulary

Fourteen words, defined once.

Under each word, the tokens a model reads instead.

Answer engine
A search service that replies with a written answer and a few cited sources, instead of a list of links.
AEO
Answer engine optimisation: making a website easy for answer engines to find, understand and quote.
GEO
Generative engine optimisation. Another name for AEO.
Passage
A paragraph, list or table that makes sense on its own: the unit answer engines retrieve and quote.
Retrieval
Searching an index for the passages that best match a question, before any answer is written.
Grounding
Basing a generated answer on retrieved sources, so that it can be checked and cited.
Citation
A link to a source shown with an answer. In AEO it does the work a ranking does in SEO.
Query fan-out
Turning one question into several related searches to gather more sources. Google’s term.
Token
The unit a language model reads: a word, part of a word or a mark of punctuation, each stored as a number.
Vector
A list of numbers that places a piece of text by its meaning, so that similar meanings sit close together.
Crawler
A program that fetches web pages for an index or a dataset. Also called a bot or a spider.
robots.txt
A file at the root of a website that tells crawlers, by name, what they may fetch.
llms.txt
A proposed Markdown file at the root of a website that points language models to its best pages.
JSON-LD
A way of writing structured data about a page in JSON, using the vocabulary of schema.org.

Anatomy

Everything a machine is sent.

The page you are reading is one of seven ways this website speaks to machines. Open each to see it exactly as a crawler does.

About this idea

¶A.1aeo.observer is an idea made by Analogue Media, a web design studio in Bath, England: a website designed first for the machines that answer questions, to see what such a website would be. It was written by Oliver Thompson, published on 22 November 2026, and every fact on it was checked against its source that week.

¶A.2The name is the job: aeo.observer watches how answer engines read the web, and is built to be read by them. To a language model the name is four tokens, “ae”, “o”, “.ob” and “server”, which is fitting for a website that serves machines first.

¶A.3The site is its own demonstration. The words are in the HTML when the page arrives, every page has a Markdown twin, the structured data describes the page as one connected graph, and the search on this page runs entirely in your browser, with nothing sent anywhere.

Analogue Media