Let AI crawlers read your site
robots.txt, server rendering and llms.txt, in plain language. Three gates decide whether an assistant can quote you at all.
The short answer
AI crawlers can only quote your site when three things are true: your robots file explicitly allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended; your page text is present in the server response rather than assembled by JavaScript in the browser; and a current sitemap plus an llms.txt file lists your important pages. Failing any one of the three makes the others irrelevant.
The Three Gates Between You and a Citation
Before any assistant can quote you, three separate things have to be true. Your robots file has to permit the AI crawlers. Your pages have to contain their text in the response itself rather than assembling it later in the browser. And your content has to be findable through a sitemap rather than only through a menu. Failing any one of the three makes the other two irrelevant.
Gate 1: Allow the Right Crawlers by Name
AI crawlers are separate from Googlebot and are often blocked by default, either by a hosting provider or by a well-meaning privacy setting. The ones that matter for citations today are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Gemini. Name each one you want to allow explicitly rather than relying on a wildcard rule, because an explicit allow survives future default changes.
There is a real decision here, not just a technical step. Allowing these crawlers means your content can be used in answers. That is the price of being cited, and for most businesses selling a service it is a bargain.
Gate 2: Serve the Text, Do Not Build It in the Browser
This is the failure that catches the most people. Many site builders and app frameworks send an almost empty page and then fill it in using JavaScript once the visitor's browser starts working. Humans never notice. Crawlers that do not execute JavaScript receive a blank page and move on.
The check takes ten seconds: fetch your page as raw source instead of viewing it in a browser and search for your own headline. If it is not there, no amount of writing will help. Server rendering fixes it permanently, and it is a platform property, not a setting you can toggle on most builders.
A site can look perfect to a visitor and be completely blank to an assistant.
Nobody gets a warning when this happens, which is why it goes unnoticed for years.
Gate 3: Give Them a Map
A sitemap listing every public page, with a single canonical domain and no stale entries, is how a crawler discovers pages that are not one click from your homepage. Orphan pages, pages that exist but are linked from nowhere, are the most common waste in a large site: real content that no crawler and no assistant will ever reach.
An llms.txt file is the newer companion to this. It is a short plain-text summary of what your site is and where the important pages live, written for models rather than browsers. Support is still uneven, but it costs almost nothing and it forces a useful exercise: stating plainly what your business does and which ten pages matter most.
Then Make the Pages Worth Quoting
Access only earns you the chance. Once the gates are open, the pages that get named are the ones that answer a narrow question completely in the first paragraph and back it with specifics a model can verify. Access is the floor, not the strategy.
Fix access once and it stays fixed. Everything else in AI visibility is ongoing work, so it is worth doing the permanent part first.
From the people who build these
The single most common cause of zero AI citations we see is not writing quality. It is a page whose headline does not exist in the raw response, because the site builds itself in the browser.
There's a better way.
Two of these three gates cannot be fixed with content work. They are decided by how your site is built and served, which is why platform choice determines your ceiling.
Not sure whether assistants can read you?
Or calculate your savings firstKeep reading
AI Agency for Content
What an AI content agency should deliver, and what you still own afterwards.
How to Get Cited in AI Search
The seven rules that decide whether a page gets named as a source.
Is ChatGPT Citing Your Website?
The monthly check that tells you where you actually stand.
SEO that destroys site builders
Why most website platforms quietly cap how well you can ever rank.
AI Search Optimization in Four Weeks
A practical month-long plan for owners rather than agencies.