Websites block AI training bots more than four times as often as AI search bots

What we found

• Training bots are blocked 5.3x as often by name

  1,744 sites name a training bot in a full block and 330 name a search or answer bot. Counting User-agent: * as well, the gap is 4.5x.

• The split holds within one company

  GPTBot is blocked on 8.0% of sites and OAI-SearchBot on 0.7%. ClaudeBot is at 7.4% and Claude-SearchBot at 0.5%.

• One template accounts for a large part of it

  35.8% of named training blocks sit in a file with the Cloudflare managed block. Without those files the shares are 8.2% and 2.1%.

• Most blockers leave AI search alone

  1,414 sites (9.9%) block training bots by name and leave every search and answer bot open. Fewer than 30 do the reverse.

• Large platforms block far more

  Among 124 large platforms we counted apart, 54.8% block a training bot by name and 32.3% a search or answer bot.

• Position makes little difference, page type more

  Sites reaching the top 3: 12.8% block training bots. Sites only at 4 to 10: 11.8%. Sites ranking mostly with comparison pages: 16.4%. With home pages: 7.9%.

Numbers to cite

Quote any figure with a link to this page. The source column names the matching rows in the CSV.

How we measured it

1. Took the list of sites

   We started from 16,399 sites seen in those top 10s. We removed 46 that belong to Seonix, its customers or visitors who analysed their site. 133 large platforms, such as social networks and marketplaces, went into a separate group, which left 16,220.

2. Asked each site once

   On 3 November 2026 we sent one request for /robots.txt and one for /llms.txt, and opened no other page. At most 20 sites at a time, a 10-second limit, up to 3 redirects, no retries. Our crawler named itself as SeonixResearch/1.0.

3. Read the rules

   We applied the standard rules for robots.txt. A bot follows its own group when one exists and the User-agent: * group otherwise. A full block means Disallow: / with nothing allowed. Files that block the root but allow some paths are counted apart.

4. Sorted the bots by job

   Training and data collection: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot. Search and answers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. We based this on what each operator publishes, where it publishes it.

5. Marked the template only when certain

   A file counts as the Cloudflare managed block only if it carries the exact comment Cloudflare documents, BEGIN Cloudflare Managed content. Copies without that comment count as a site's own rules.

6. Applied a publication floor

   Aggregates only, no site named, no count under 30. Raw answers stay on our side and are not published.

What limits this data

• Not the whole web. The list comes from commercial searches chosen for 23 sites that use Seonix, and from DuckDuckGo. Sites that rank elsewhere may behave differently.
• A request, not a fact. Bots may ignore robots.txt, and a site can also block them at the firewall, which this study cannot see. We sent no requests as an AI bot.
• Unreachable is not the same as open. 496 sites refused our request for the file, and that says nothing about their rules for AI bots. Those sites and the other failures are outside every blocking share.
• A missing file means no block, but we keep it apart. 448 sites had no file and 479 returned a web page instead. They are also outside the shares, so counting them would lower every share slightly.
• A template says little about intent. We can see that a file carries the Cloudflare block, but not who turned it on or why.
• Hosts, not companies. A subdomain counts as its own site, and 165 requests ended on another host after a redirect.
• Labels overlap. Google-Extended also covers grounding in Gemini Apps and does not affect Google Search. Amazonbot serves several purposes, including AI training. Page types are labels read from URLs and titles.
• One moment. This is a single crawl on one day, so it shows no trend.

Also measured: llms.txt

3,604 of the 14,875 sites that answered clearly (24.2%) serve a valid llms.txt: at least 100 characters, a heading and a link. It is an optional convention. We did not measure whether any AI engine uses it.