ROBOTS.TXT TESTER · FREE TOOL
Is that URL allowed or blocked?
Paste your robots.txt, pick a crawler — Googlebot, GPTBot, ClaudeBot, PerplexityBot — and a URL. See instantly whether it's allowed, and exactly which rule decides. Uses the standard longest-match logic.
How robots.txt rules are evaluated
A crawler looks for the group of rules matching its user-agent (or the * group if none match). Among that group's Allow and Disallow rules, the one with the longest matching path wins; if an Allow and a Disallow match with equal length, Allow wins. An empty Disallow: means "allow everything." This tester applies those standard rules, including * and $ wildcards.
Free utility for guidance. Implements the widely-adopted robots.txt matching rules; individual crawlers may differ slightly. Always confirm critical rules with the crawler's own documentation or Search Console.
FAQ
How do I know if a page is blocked?
Paste your robots.txt, pick the bot, enter the URL — you'll get an ALLOWED/BLOCKED verdict and the deciding rule.
Does this cover AI crawlers?
Yes — GPTBot, ClaudeBot, PerplexityBot and others follow robots.txt; test any of them here.