How to allow ChatGPT, Claude, Gemini and Perplexity crawlers in robots.txt

Updated 2026-10-27

To be visible in AI answers, do not block the AI search and user crawlers in robots.txt: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity). Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a separate choice: blocking them limits model training use but is a policy decision, not a requirement.

The main AI crawlers

OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results), ChatGPT-User (fetches when a user asks).
Anthropic: ClaudeBot (training), Claude-SearchBot (search quality), Claude-User (fetches when a user asks).
Perplexity: PerplexityBot (search index), Perplexity-User (user-initiated fetches).
Google: Google-Extended is a control token for Gemini use of your content; it is not a separate crawler and does not affect Google Search ranking.
Apple: Applebot-Extended controls use of content for Apple AI features.

A permissive robots.txt template

User-agent: *
Allow: /
Disallow: /api/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Common mistakes

A blanket "User-agent: * / Disallow: /" left over from staging.
A CDN or firewall bot-protection rule that blocks AI user-agents even though robots.txt allows them.
Blocking GPTBot and assuming that also blocks ChatGPT search. They are different agents.
Content that only appears after JavaScript runs. Many AI fetchers do not execute scripts.

Check it in 30 seconds

The free LLMSpotter scan tests your robots.txt for AI crawler access, along with five other readiness checks, and gives a score out of 120.

See how AI crawlers read your site

Run the free scan

See a sample AI Visibility Report