Which AI Crawlers to Allow in robots.txt (2026 Guide)

Short answer: If you want AI answer engines to cite your brand, you generally want to allow the AI crawlers that build search and answer indexes — like OAI-SearchBot (ChatGPT search), PerplexityBot, and ClaudeBot — in your robots.txt. Blocking them removes you from the pool of content AI can reference, which quietly kills your AI visibility. The one nuance worth a decision is training crawlers (like GPTBot and CCBot): allowing them lets your content be used for model training, which some brands accept and some don't.

TL;DR — Key Takeaways

• Blocking AI crawlers hurts AI visibility. If a bot can't read your site, the engine can't cite you.
• Two kinds of bots: training crawlers (feed model training) and search/user crawlers (fetch pages to build answers or respond to a user's live query). The distinction matters.
• Allow the ones that build answers: OAI-SearchBot, PerplexityBot, ClaudeBot, and the user-triggered fetchers.
• Decide on training crawlers: GPTBot and CCBot (via Common Crawl) are about training use — a brand/legal choice, not a visibility one.
• Google-Extended is a control token, not a separate crawler, and does not affect Google Search ranking.
• Verify your file loads and check server logs to confirm bots are actually reaching you. AlpynAI can audit this for you.

What are AI crawlers, and why do they matter?

AI crawlers are automated bots that AI companies use to read the open web. They do a few different jobs: - Training crawlers gather content to train or update large language models. - Search/index crawlers build the indexes that answer engines search over when they compose a response. - User-triggered fetchers grab a specific page in real time because a user asked the assistant about it (for example, pasting a URL into ChatGPT).

Here's why this matters for your brand: you can only be cited from content an AI is allowed to read. If your robots.txt blocks the crawler that builds Perplexity's index, Perplexity has nothing of yours to cite. AI visibility starts with access.

What are the AI crawler user-agent tokens?

These are the current, verified user-agent tokens by provider. Use these exact names in robots.txt.

A few clarifications that trip people up: - Google-Extended is not a crawler. It's a control token you use in robots.txt to opt in or out of having your content used for Gemini's training and grounding. It does not crawl separately, and setting it does not change your Google Search ranking. Google Search uses Googlebot, which is governed independently. - Deprecated Anthropic tokens: older guides mention anthropic-ai and Claude-Web. Both are deprecated — use ClaudeBot and Claude-User instead. - CCBot belongs to Common Crawl, a nonprofit whose dataset feeds many AI training pipelines. Blocking it removes you from a huge amount of downstream AI training data.

What's the difference between training crawlers and user fetches?

This is the single most useful distinction for making a smart decision:

• Training crawlers (GPTBot, ClaudeBot, CCBot) collect content to train future models. Allowing them means your content may contribute to how models "know" your brand over the long term. Blocking them is a legitimate choice if you don't want your content used for training — but it's a brand/legal/IP decision, and it may reduce how well models understand you over time.
• Search/index crawlers (OAI-SearchBot, PerplexityBot) build the live indexes answer engines search when responding. Blocking these directly reduces your chance of being cited in AI answers right now.
• User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) fetch a page because a real user asked the assistant to look at it. Blocking these means a user who explicitly wants to pull in your page can't. That's usually the last thing you want to block.

The practical implication: even brands that are cautious about training usually still want to allow the search and user crawlers, because those are what put you into answers.

What does blocking AI crawlers cost me?

Blocking is the silent killer of AI visibility. Concretely: - You disappear from answer engines. If PerplexityBot and OAI-SearchBot can't read you, you won't be cited in Perplexity or ChatGPT search results — no matter how well you rank in Google. - Models' understanding of your brand goes stale. Blocking training crawlers means new information about you never enters the model's world; it may keep describing an old version of you, or default to competitors. - You can't undo a citation you never earned. Visibility compounds. Time spent blocked is visibility permanently forgone.

The tradeoff on the other side is real but narrower: allowing training crawlers means your public content can be used to train models. If that's a genuine IP concern for you, block the training tokens specifically while keeping the search/user tokens open.

What is a recommended robots.txt for AI visibility?

Here's a robots.txt that allows the AI crawlers so you can be found and cited. This is the recommended default for most brands that want maximum AI visibility:

# Allow AI crawlers for maximum AI visibility

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

# Google (Gemini training/grounding control token)
User-agent: Google-Extended
Allow: /

# Common Crawl (feeds many AI datasets)
User-agent: CCBot
Allow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you want to allow answer/search crawlers but opt out of training, block only the training-oriented tokens while keeping the rest open:

# Allow answer engines, opt out of training

# Opt out of training use
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Keep search + user fetchers open so you can still be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Note that ClaudeBot is Anthropic's training crawler; if you're strictly opting out of all training, you'd Disallow it too and rely on Claude-User for user-triggered fetches. Choose based on how strict your training stance is.

How do I verify AI crawlers can access my site?

Adding rules isn't the same as them working. Verify:

1. Confirm the file loads. Visit https://yourdomain.com/robots.txt in a browser and read it. It must be at the domain root.
2. Check syntax. Each User-agent block needs its own Allow/Disallow lines. A stray Disallow: / under User-agent: * can block everything.
3. Watch your server logs. Look for the user-agent strings (GPTBot, PerplexityBot, ClaudeBot, etc.) hitting your site. Real crawler visits confirm access (exact log formats vary by host and server).
4. Test being cited. Ask ChatGPT, Perplexity, and Claude about your brand and category and see whether you appear. If you were previously blocked, allow it, then re-check after crawlers have had time to re-index.
5. Re-audit after site changes. A CMS migration or a new security/CDN rule can silently reintroduce blocks.

How can AlpynAI help?

AlpynAI can audit whether you're accidentally blocking AI crawlers and generate a corrected robots.txt (along with other fixes like schema and llms.txt). It then measures how ChatGPT, Claude, Perplexity, and Gemini actually reference your brand and reports an AI Citation Score (0–100), so you can confirm that opening access translates into real citations. Start with a free brand check at alpynai.com — no credit card required.

FAQ

Which AI crawlers should I allow in robots.txt?

For maximum AI visibility, allow the search/index and user-triggered crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, and Claude-User. Whether you also allow the training crawlers (GPTBot, CCBot) and the Google-Extended control token is a separate brand decision.

Does blocking AI crawlers hurt my AI visibility?

Yes. If a crawler can't read your site, the engine it feeds can't cite you. Blocking search and user crawlers directly removes you from AI answers, regardless of how well you rank in traditional search.

Does Google-Extended affect my Google Search ranking?

No. Google-Extended is a control token for opting in or out of Gemini's training and grounding use of your content. It is not a separate crawler and does not affect Google Search ranking, which is governed by Googlebot.

What's the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's training crawler; allowing it lets your content be used to train models. OAI-SearchBot builds the ChatGPT search index used to compose answers. If you want to be cited but avoid training use, allow OAI-SearchBot and consider blocking GPTBot.

Are anthropic-ai and Claude-Web still valid?

No. Both anthropic-ai and Claude-Web are deprecated. Use ClaudeBot (training) and Claude-User (user-triggered fetches) instead.