AI Crawler Checker
Enter a domain to see which AI crawlers its robots.txt allows and which it blocks. The check covers the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Copilot and Meta AI, and tells you what blocking each one actually costs — because blocking a training crawler and blocking a search crawler have very different consequences.
The same check also reports whether the site publishes an llms.txt and which sitemaps its robots.txt declares.
Updated August 2026
How to use it
- 1
Enter the domain
Type the site address — example.com is enough, the scheme is optional. The check reads the site's public robots.txt.
- 2
Read the three groups
AI search crawlers decide whether you can be cited in answers. Training crawlers decide whether your text trains models. Classic index crawlers decide whether you exist in Google and Bing at all.
- 3
Check the rule that decided it
Every blocked crawler shows the exact robots.txt line responsible, and whether it came from its own group or the catch-all * group.
How to read the result
robots.txt is a request, not a lock. It states what a site asks crawlers to do. Well-behaved crawlers obey it; one that ignores it will not show up here, and no public tool can prove otherwise.
"Not mentioned" means allowed: robots.txt permits everything it does not explicitly forbid. A site with no robots.txt at all is fully crawlable.
Blocking a training crawler (GPTBot, Google-Extended, ClaudeBot) does not remove you from AI answers — that depends on the search crawlers. Blocking Googlebot does remove you from both Google Search and AI Overviews.
Frequently asked questions
- Should I block AI crawlers?
- It depends on what you sell. If discovery matters more than content protection, keep the AI search crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) allowed. Blocking training crawlers while allowing search ones is a common middle ground.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot collects data for training OpenAI models. OAI-SearchBot builds the index ChatGPT search cites from. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot keeps you out of answers.
- Does Google-Extended affect my Google rankings?
- No. Google-Extended only governs Gemini training and grounding. Normal Search crawling and ranking are controlled by Googlebot.
- Does this tool store the sites I check?
- No. The server fetches the public robots.txt, parses it, returns the result and keeps nothing.