A robots.txt Template for Managing AI Crawlers

Your robots.txt file is the shortest document on your site and one of the most misunderstood. It is also the first place you can tell an AI crawler what it may and may not read. This guide gives you a working template and the reasoning behind every line.

What an AI Crawler Actually Is

An AI crawler is a bot operated by a model provider that fetches pages to read, index, or train from them. Unlike a search crawler that mainly builds a ranking index, an AI crawler may feed your text into a model's knowledge or use it to ground generated answers. The bot identifies itself through a user-agent string, and robots.txt is the standard way to address that string.

You do not control what a model already learned from other sources. You do control whether a specific bot can fetch your live pages going forward. That distinction matters, and the template below is built around it.

Why This Matters for GEO

Being cited by an AI engine almost always starts with being crawled. If you block every AI bot, you remove the most direct path for a model to read your live, current pages and decide they are worth quoting. That is why a blanket block is rarely the right first move for a brand that wants visibility.

The template above defaults to allowing the major crawlers while letting you block the ones you distrust. That balance keeps you visible where it helps and private where it matters. Revisit the list every quarter as new bots appear and as your own policy on content reuse changes.

The Template

Copy this file, save it as robots.txt at the root of your domain, and edit the Allow and Disallow lines to match your policy. Every line is commented so you can see what each rule does.

# AI crawler management template
# Edit the Allow/Disallow lines to match your policy.
# Save this file at: https://www.yourdomain.com/robots.txt

# Allow standard search crawlers (they also power some AI Overviews)
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Allow OpenAI's crawler if you want ChatGPT to read and cite you
User-agent: GPTBot
Allow: /

# Allow Google's AI-specific crawler
User-agent: Google-Extended
Allow: /

# Allow Perplexity's crawler
User-agent: PerplexityBot
Allow: /

# Allow Anthropic's crawler
User-agent: ClaudeBot
Allow: /

# Allow Apple's AI crawler
User-agent: Applebot-Extended
Allow: /

# Block a crawler you do not want reading your pages
User-agent: CCBot
Disallow: /

# Block another AI crawler by name
User-agent: OAI-SearchBot
Disallow: /

# Help crawlers find everything
Host: www.yourdomain.com
Sitemap: https://www.yourdomain.com/sitemap.xml

How to Allow the Crawlers You Want

If your goal is to be cited, allow the crawlers that ground answers in live web content. OpenAI's GPTBot, Google's Google-Extended, Perplexity's PerplexityBot, and Anthropic's ClaudeBot are the names most teams start with. Add each as its own User-agent block with Allow: /.

Keep the standard search crawlers allowed too. Googlebot and Bingbot still power the traditional results that often sit beside AI answers, and blocking them can cost you more than you gain.

How to Restrict Specific Bots

To block a bot, add a block with Disallow: /. Use the exact user-agent string the provider publishes. Common examples you might block include CCBot, which is associated with broad web-scale crawling, or any bot whose use of your content you have decided against.

Remember that robots.txt is a request, not a firewall. A well-behaved bot honors it; a malicious one ignores it. For sensitive content, access control at the server or application layer is the real safeguard.

Testing Your File

After you publish, verify the file is reachable and parseable. Most search consoles include a robots.txt tester that shows which rules apply to a given user agent and URL. Run a check for each bot you explicitly allowed or blocked to confirm the rule behaves as written.

Also confirm your sitemap is listed and reachable. A crawler that is allowed but cannot find your pages through the sitemap or internal links will still miss content.

Common Mistakes

The first mistake is a typo in the user-agent name, which silently fails the rule. The second is using Disallow: / at the top of the file without a specific User-agent, which blocks every crawler, including the ones you wanted. The third is forgetting that changes take effect only after the bot next visits, so allow a crawl cycle before judging results.

A Block-By-Default Variant

If your priority is limiting reuse rather than maximizing citation, invert the logic: block all AI crawlers, then allow only the few you trust. Use this variant when legal, competitive, or licensing reasons outweigh the benefit of being cited.

# Restrictive variant: block AI crawlers by default, allow only chosen ones

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /

The final catch-all rule blocks every user agent not named above. Use it only if you are comfortable also blocking standard search crawlers, since the wildcard catches them too. If you want search traffic but not AI reuse, name Googlebot and Bingbot in their own Allow blocks before the catch-all so they stay permitted.

Reading Logs to Confirm Compliance

A robots.txt rule is only as good as the bots that honor it. Pull your server logs and filter by user agent to see whether GPTBot, ClaudeBot, and the others are hitting your pages with 200 responses, or being turned away with 403s where you intended. A mismatch usually means a typo in the agent name or a rule-order problem.

Logs also reveal bots you forgot to address. If you see an AI crawler you did not name in the file, decide deliberately whether to allow or block it rather than leaving the gap implicit. The goal is a policy you wrote, not one you drifted into.

Combining robots.txt With Other Controls

robots.txt is one layer, not the whole privacy story. Pair it with a clear content licensing statement on your pages, so both bots and humans understand your intent. Pair it with server-side access rules for anything truly sensitive, because a determined crawler will ignore a text request.

You can also use page-level meta tags and HTTP headers as controls that complement the site-level file. The point is defense in depth: state your policy in robots.txt, enforce it technically where it counts, and revisit both when a new crawler or a new business need appears.

Frequently Asked Questions

Does blocking a crawler remove my content from existing AI answers?

No. robots.txt only affects future crawls. Content a model already ingested or that it reaches through another source is not removed by this file. Treat it as forward control, not erasure.

Should I block all AI crawlers by default?

That depends on your goal. If you want to be cited and recommended, blocking everything works against you. If your priority is limiting how your content is reused, a selective block list is the more precise tool.

Where exactly does the robots.txt file go?

It must sit at the root of your domain, served at the path /robots.txt, so any crawler can fetch it without authentication. A file buried in a subfolder will not be read as your site policy.