Key takeaways
- Search and retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) drive visibility in AI answers.
- Training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, CCBot) can be blocked without leaving search answers.
- Google-Extended does not affect inclusion in Google Search; AI Overviews draw on content crawled by Googlebot.
- A crawler that matches its own User-agent group ignores the * group — repeat shared Disallow rules.
- CDN and firewall bot rules can block AI crawlers even when robots.txt allows them.
The AI crawler map
| User agent | Company | Purpose | Affects AI visibility? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Yes |
| GPTBot | OpenAI | Content that may train foundation models | Training only |
| ChatGPT-User | OpenAI | User-initiated fetches; robots.txt may not apply | Per request |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | Product token: Gemini training and grounding in Gemini Apps / Vertex AI | Gemini only; not Search | |
| Bingbot | Microsoft | Bing Search, which grounds Copilot | Yes |
| PerplexityBot | Perplexity | Indexing for Perplexity search results | Yes |
| Perplexity-User | Perplexity | User-initiated fetches; generally ignores robots.txt | Per request |
| Claude-SearchBot | Anthropic | Search indexing for Claude | Yes |
| ClaudeBot | Anthropic | Training data collection | Training only |
| Claude-User | Anthropic | User-directed retrieval | Per request |
| Applebot-Extended | Apple | Opt-out token for Apple model training | Training only |
| CCBot | Common Crawl | Open web dataset widely used for training | Training only |
Decide your policy first
There are two separate questions. Do you want to appear in AI answers? For most companies the answer is yes, so search and retrieval crawlers should be allowed. Do you want your content used to train models? That is a business and legal decision. Many marketing sites allow training to strengthen how models understand their brand; publishers with licensable content often block it. Either way, you can choose independently.
A robots.txt you can adapt
This template allows search and retrieval crawlers, blocks common training crawlers, and keeps private paths closed for everyone. Delete the training block if you want to allow training.
# Search & answer engines — allow, but keep private paths closed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /checkout/
# Model training — your decision
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xmlFive gotchas we see constantly
- Group inheritance. Under RFC 9309, a crawler follows the most specific group that names it and ignores
User-agent: *. If you give Googlebot its own group, repeat your shared Disallow lines there. - CDN and firewall rules. Bot-management features can block or challenge AI crawlers regardless of robots.txt. Check your CDN’s AI crawler settings and logs.
- Managed robots.txt. Some CDNs can prepend their own directives to your file. Fetch your live robots.txt to see what crawlers actually receive.
- JavaScript-only content. Allowing a crawler is pointless if the answer lives in client-side rendered content it does not execute.
- Subdomains. Each subdomain needs its own robots.txt at its root.
What about content signals and llms.txt?
Newer conventions let sites express preferences beyond crawl/no-crawl — for example, machine-readable signals about search, AI input and AI training use, and the llms.txt proposal for a concise, model-friendly site map. They are useful additions, but support varies by company. Treat robots.txt plus crawler access as the foundation, and these as enhancements.
Verify it worked
- Fetch
/robots.txtfrom outside your network and confirm the live content. - Watch server or CDN logs for OAI-SearchBot, PerplexityBot and Claude-SearchBot requests over the following days.
- Run our free AI-readiness check — it parses your robots.txt for 14 AI user agents.