How to Configure robots.txt for AI Crawlers (GPTBot, ClaudeBot, PerplexityBot & More)

Name the retrieval crawlers explicitly: as of 2026 that means OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, PerplexityBot, and Google-Extended. Think in two groups. Retrieval crawlers fetch content to answer a live user question and can cite you in the response — these are the ones you must allow to be eligible for AI answers. Training crawlers ingest content to train models. Per Google Search Central, robots.txt is the standard way to tell these crawlers which paths they may fetch. The stakes are real: ChatGPT alone reportedly handles on the order of 2.5 billion prompts per day — a figure cited by Genrank without a named primary source, so treat it as indicative of scale rather than a precise measure.

OpenAI is the common trap: it operates three distinct crawlers — GPTBot (training), ChatGPT-User (browsing and retrieval), and OAI-SearchBot (search retrieval) — so a robots.txt that only names GPTBot silently misses the other two. Anthropic uses ClaudeBot and an older anthropic-ai token; the exact relationship between those tokens is not clearly documented, so treat them as separate entries rather than assuming they are equivalent.

Google-Extended behaves differently from the rest: it is a control token, not a separate HTTP user-agent. Actual crawling is performed under existing Google user-agent strings, which means you will not see Google-Extended traffic isolated in your server logs even though the robots.txt token still governs whether your content can be used for Gemini and AI Overviews. Training-only crawlers to be aware of include CCBot (Common Crawl), Bytespider (ByteDance), and Applebot-Extended.

By default, AI crawlers are not blocked — they will crawl your site freely unless you explicitly disallow them in robots.txt. So doing nothing means you are allowing every compliant AI bot, which is usually what you want for visibility.

robots.txt is also not a privacy or invisibility tool. A page you disallow can still be indexed by Google if it is linked from other sites, so disallowing a URL does not guarantee it stays out of results. And opting out of some AI uses is not always a robots.txt job: Bing, for example, uses its crawlers for both search and AI training, and opting out of the AI-training use requires a page-level meta tag rather than a robots.txt directive.

Finally, remember the hard limitation: compliance is voluntary. Well-behaved crawlers honor your rules, but non-compliant scrapers routinely ignore robots.txt, spoof user-agent strings, or use residential IPs to look like ordinary browsers — so robots.txt is the right tool for managing compliant AI engines, not for stopping determined scraping.

A permissive, citation-friendly configuration names each of the 6 retrieval bots explicitly and allows it. In practice that is a block of User-agent: GPTBot followed by Allow: /, then the same pattern repeated for OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended, finished with a single Sitemap: https://yourdomain.com/sitemap.xml line so crawlers can discover your pages.

If you want to remain citable while opting out of model training, keep the Allow blocks above for the retrieval bots and add separate blocks that Disallow: / for training-only crawlers such as CCBot, Bytespider, and Applebot-Extended. Be deliberate about this split: the retrieval bots are the ones that can send you AI citations, so blocking them removes you from that engine's answers entirely.

Keep the rules simple and explicit. A short, well-labeled robots.txt that names each AI user agent is easier to audit and less error-prone than clever wildcard logic.

You can allow retrieval bots — so you remain eligible for citation — while disallowing training-only crawlers like CCBot, Bytespider, and Applebot-Extended. This is a values and bandwidth decision, not a ranking optimization, because there is no evidence that blocking training crawlers improves or harms your citation outcomes, so decide based on your stance on having your content used for model training.

One caveat on enforcement: the compliance behavior of some training crawlers, including Bytespider and CCBot, is not reliably confirmed, so a Disallow directive is a request that compliant bots honor rather than a guarantee. Community projects such as ai-robots-txt/ai.robots.txt maintain curated lists of AI user-agent strings if you want a maintained reference for which tokens to target.

If your priority is simply being visible in AI answers, the safe default is to allow the retrieval bots and leave training-crawler decisions for later.

The most damaging mistake is case sensitivity: user-agent tokens are case-sensitive, which means a rule for gptbot does not match GPTBot and will be ignored by compliant crawlers. Always copy the exact casing from each vendor's documentation.

The next most common errors are naming only GPTBot (and missing ChatGPT-User and OAI-SearchBot), using an over-broad wildcard Disallow that accidentally catches AI bots, relying on robots.txt to hide content, and forgetting the Sitemap directive. Each of these quietly reduces your eligibility for AI answers.

Watch your CDN and WAF too. Cloudflare's managed robots.txt and AI block rule (part of Super Bot Fight Mode) can override a custom Allow rule for a bot like GPTBot unless you explicitly disable the managed block first — so a robots.txt that looks permissive can still be overridden at the edge. Cloudflare also offers a managed Content Signals Policy that lets you set preferences such as ai-train=no without hand-editing robots.txt.

Run these 4 quick checks. First, open your /robots.txt and confirm each retrieval bot — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended — is allowed (or at least not disallowed) with exact casing. Second, check your CDN or WAF for managed AI-blocking rules that could override the file. Third, inspect your response headers for an X-Robots-Tag that might restrict access. Fourth, confirm your Sitemap line is present and the sitemap is reachable.

Doing this by hand is fiddly, especially the per-bot and edge-rule checks. Our free AI crawlability checker runs these checks for you across the major AI crawlers and returns a pass or fail per bot with specific fixes.

For the wider context — rendering, schema, and how engines actually choose sources — read the complete AI crawlability guide.

Configuring robots.txt for the major AI crawlers comes down to 5 points: allow the 6 retrieval bots, mind OpenAI's 3 separate crawlers, respect case-sensitive tokens, accept that robots.txt is voluntary, and watch for edge rules that override it.