How to Block AI Bots in Robots.txt

If you want search engines to keep crawling your site while AI training bots are discouraged, robots.txt is the simplest public signal you can publish. The key is to give each bot its own rule block rather than hiding everything behind one vague setting.

Example AI bot block list

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

You can pair that with a normal all-bots rule or keep your existing search engine setup unchanged.

Allow search engines, block AI bots

User-agent: *
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

This structure makes your intent clearer: search crawlers can proceed, but the listed AI bots should not crawl the site.

Limitations to understand

Generate an AI-blocking robots.txt file

The generator includes a one-click AI bot preset so you can start from the common block list, add your sitemap, and adjust any extra rules before publishing.

Open the AI Bot Preset