AI crawlers are bots that fetch your pages to train large language models and to ground the answers generated by AI search engines. You control them the same way you control search crawlers: with user-agent rules in robots.txt. Adding a Disallow for a token like GPTBot or Google-Extended blocks that specific bot — while leaving Googlebot, and your search rankings, completely untouched.
How are AI crawlers different from search crawlers?
A traditional search crawler like Googlebot fetches pages to build a search index you rank in. AI crawlers do one of two different jobs, and the distinction determines your strategy. Training crawlers (GPTBot, ClaudeBot, CCBot) collect text to train future models — content they take is absorbed into model weights, not linked back. Retrieval crawlers (OAI-SearchBot, PerplexityBot) fetch pages in real time to cite as sources inside a generated answer, usually with an attribution link. Blocking the first type protects your content; blocking the second type removes you from AI answers.
✦ Insight
Insight: Google-Extended is the single most misunderstood token. Disallowing it stops Google from using your content to train and ground Gemini — but it has zero effect on how Googlebot crawls you, how you rank in Search, or whether you appear in AI Overviews. AI Overviews are built from Google's standard search index, so you cannot opt out of them with Google-Extended.
Which AI crawlers should you know about?
These are the major AI crawlers most sites will see in their access logs as of 2026. Each is identified by a user-agent token you can target in robots.txt:
- GPTBot — OpenAI's crawler for training its models. Respects robots.txt. Blocking it removes your content from future training runs.
- OAI-SearchBot — OpenAI's crawler for surfacing and citing pages in ChatGPT search results. Distinct from GPTBot; allow it if you want to be cited.
- Google-Extended — a robots.txt token (not a separate bot) controlling whether your content trains and grounds Gemini. Does not affect Googlebot, Search, or AI Overviews.
- ClaudeBot — Anthropic's crawler for training Claude. Respects robots.txt.
- PerplexityBot — Perplexity's crawler for indexing pages it cites as sources in its answers.
- CCBot — Common Crawl's crawler. Its open dataset feeds many LLM training pipelines, so blocking CCBot has broad downstream reach.
- Bytespider — ByteDance's crawler. Historically aggressive, and its robots.txt compliance has been reported as inconsistent.
How do you block or allow AI crawlers in robots.txt?
Each crawler gets its own User-agent block with Allow or Disallow rules, exactly like Googlebot. The example below blocks the training crawlers while keeping the retrieval crawlers open — a common configuration for a brand that wants to protect its content from training but still be cited in AI answers:
# robots.txt — AI crawler control
# Block OpenAI model training, but allow its search crawler
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
# Opt out of Gemini training/grounding (does NOT affect Google Search)
User-agent: Google-Extended
Disallow: /
# Block Anthropic and Common Crawl training
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Allow Perplexity so you remain eligible for citation
User-agent: PerplexityBot
Allow: /⚠️ Warning
robots.txt is a voluntary standard. Well-behaved crawlers like GPTBot and ClaudeBot honour it, and Google respects the Google-Extended token, but nothing technically enforces it. Crawlers that ignore robots.txt can only be stopped at the server, CDN, or firewall level — by blocking their user-agent string or IP ranges. Treat robots.txt as your first line of control, not a guarantee.
Should you block AI crawlers or allow them?
This is a strategic decision, not a technical default. Blocking training crawlers protects your original content and proprietary data from being absorbed into models without attribution or compensation. But blocking the retrieval crawlers that power AI answer engines removes you from the surfaces where a growing share of informational traffic now lands. The two goals conflict, and the right answer depends on your business model.
- Publishers monetising original content (news, research, premium data): blocking training crawlers (GPTBot, ClaudeBot, CCBot) protects your primary asset from uncompensated reuse.
- Brands pursuing GEO visibility: allow the retrieval crawlers (OAI-SearchBot, PerplexityBot) so you stay eligible for citation inside AI answers.
- Most marketing and lead-gen sites: allow everything — the citation and referral upside outweighs the training concern, because your goal is discovery, not content licensing.
- Remember that a Disallow is not retroactive: robots.txt stops future crawling, but content already absorbed into a trained model cannot be recalled.
Does blocking AI crawlers hurt your Google rankings?
No — with one critical caveat. Blocking a training crawler like GPTBot or ClaudeBot, or disallowing the Google-Extended token, has no effect on your Google Search rankings, because none of those tokens control Googlebot — the crawler that actually builds Google's search index. The only way to damage your rankings is to accidentally block Googlebot itself. And because AI Overviews are generated from the standard search index, you cannot remove yourself from AI Overviews via Google-Extended; you would have to deindex from Search entirely, which almost no site should do.
How do you verify your AI crawler rules are working?
Two checks confirm your rules are live. First, open yourdomain.com/robots.txt in a browser and confirm each User-agent block is present and served with a 200 status — a common mistake is editing a local file that never deploys. Second, watch your server access logs for the AI user-agent strings: after you deploy a Disallow, the matching bot's requests should taper off within days. Note that Google Search Console's robots.txt report only tests Google's own tokens, so third-party AI crawlers won't appear there — the server logs are your source of truth for those.
💡 Tip
Practice this in the game: Chapter 4-1 (The Generative Ghost Town) drops you into a site that accidentally blocked every AI crawler — retrieval bots included — and vanished from AI answers overnight. You'll audit the robots.txt, separate the training crawlers from the search crawlers, and rebuild AI visibility without giving away the content you meant to protect.