What this checker reviews
It reads robots.txt groups and checks whether major AI search, training, user-triggered, and control-token agents are explicitly covered or only inherit general crawler rules.
AI crawler access
Robots.txt AI Crawler Checker
Paste your robots.txt and see whether major AI search, training, and user-triggered crawler tokens are allowed, blocked, or unclear before AI systems discover your pages.
Paste and check instantly. Your robots.txt content stays on this device.
Ready to check AI crawler rules
Paste robots.txt to inspect OpenAI, Anthropic, Google, Perplexity, and Common Crawl access signals.
Why AI crawler rules matter
Robots.txt can affect whether AI answer engines, browsing agents, model-training crawlers, and dataset crawlers can reach public pages for discovery or citation.
How to use the result
Resolve mixed rules first, then decide whether each AI crawler should be allowed for visibility or blocked for content-control reasons.
Related AI search tools
FAQ
Which AI crawlers does this check?
It checks OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, PerplexityBot, and CCBot, plus the fallback rule for all crawlers.
Should I allow GPTBot and ClaudeBot?
Allow search-focused agents when AI search visibility matters. Treat model-training agents separately when content reuse control is more important than training-data inclusion.
Is Google-Extended the same as Googlebot?
No. Google-Extended is a robots.txt control token, not a separate HTTP crawler user agent, and it does not control normal Google Search inclusion.
Can robots.txt guarantee AI systems will not use my content?
Robots.txt is an access policy for cooperative crawlers, not a legal or technical guarantee against every content use.