AI CRAWLER CONTROL · FREE TOOL
Decide which AI bots can use your site.
GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more crawl the web to train models and answer questions. Choose block or allow for each, and generate the exact robots.txt rules — copy & paste, no signup.
robots.txt
Paste into (or merge with) yourdomain.com/robots.txt at your site root.
Why control AI crawlers?
Two different kinds of bots visit your site. Training crawlers (GPTBot, Google-Extended, CCBot, Bytespider) collect content to train AI models — block these if you don't want your work used that way. AI search/answer bots (OAI-SearchBot, PerplexityBot) fetch pages to cite in AI answers — usually worth allowing, since being cited is free visibility in the new AI search world.
robots.txt is a voluntary standard. Major AI providers document and honor it, but it isn't a hard technical block. This is a free utility; the bot list reflects publicly documented user-agents and evolves over time.
FAQ
How do I block GPTBot or ClaudeBot?
Disallow their user-agent in robots.txt. This tool generates those rules — pick "block" and paste the output at your site root.
Does robots.txt really stop AI?
Major AI companies honor robots.txt for their crawlers. It's voluntary, not a hard block, but it's the accepted way to state your preference and the big players respect it.
Block or allow?
Block training bots if you don't want your content training models; allow AI search bots if you want to be cited in AI answers.