AI search · GEO & AEO

AI Crawler Checker

Enter a domain to see which AI crawlers its robots.txt allows and which it blocks. The check covers the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Copilot and Meta AI, and tells you what blocking each one actually costs — because blocking a training crawler and blocking a search crawler have very different consequences.

The same check also reports whether the site publishes an llms.txt and which sitemaps its robots.txt declares.

FreeNo signupChecks the live pageUpdated October 2026

AI Crawler Checker

Our server fetches the public page you name and keeps no copy of it.

Seonix Site Health dashboard: an overall score of 91 out of 100, SEO, technical, speed and AI search scores, and the list of issues to fix — Site Health checks every page for SEO, technical, speed and AI-search issues and marks which ones Seonix fixes for you.

How it works

How to use it

01
Enter the domain
Type the site address — example.com is enough, the scheme is optional. The check reads the site's public robots.txt.
02
Read the three groups
AI search crawlers decide whether you can be cited in answers. Training crawlers decide whether your text trains models. Classic index crawlers decide whether you exist in Google and Bing at all.
03
Check the rule that decided it
Every blocked crawler shows the exact robots.txt line responsible, and whether it came from its own group or the catch-all * group.

Honest notes

How to read the result

robots.txt is a request, not a lock. It states what a site asks crawlers to do. Well-behaved crawlers obey it; one that ignores it will not show up here, and no public tool can prove otherwise.

"Not mentioned" means allowed: robots.txt permits everything it does not explicitly forbid. A site with no robots.txt at all is fully crawlable.

Blocking a training crawler (GPTBot, Google-Extended, ClaudeBot) does not remove you from AI answers — that depends on the search crawlers. Blocking Googlebot does remove you from both Google Search and AI Overviews.

Seonix does this on autopilot

Being crawlable is step one. Being cited is the job.

Seonix finds the questions your buyers ask search engines and AI assistants, publishes pages that answer them on your site, and shows where your brand appears in Google results and ChatGPT answers.

Analyze my site free

For your stack

SEO on autopilot for the site you have

Seonix researches, writes and publishes articles — and keeps the technical side in check — wherever your site lives.

WordPress sites

The free Seonix plugin connects in one click: articles publish as native posts, and SEO fixes and llms.txt are handled for you.

WordPress SEO automation

SaaS products

Your AI coding agent connects your app to Seonix from one prompt; keyword-backed articles then publish to your own domain on schedule.

SaaS SEO on autopilot

Custom-built websites

No WordPress needed: your site exposes one small endpoint and articles arrive as native pages in your own design.

SEO for custom websites

Also onShopify Webflow Wix Ghost

Frequently asked questions

Should I block AI crawlers?

It depends on what you sell. If discovery matters more than content protection, keep the AI search crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) allowed. Blocking training crawlers while allowing search ones is a common middle ground.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects data for training OpenAI models. OAI-SearchBot builds the index ChatGPT search cites from. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot keeps you out of answers.