# Fact sheet: robots.txt census, top 500 domains

Snapshot 2026-07-29. Sample: Tranco list ZJL5G, top 500 domains. Python 3.14.3. Source of truth: `python run.py` on the shipped snapshot. One fetch per domain (https, single http fallback), honest User-Agent, failures recorded as data. Total transfer 8.7 MB.

## Reachability (all n/500)

- Valid robots.txt: 235/500, 47.0%
- HTTP 200 but not a robots.txt (HTML or junk at the robots address): 36/500, 7.2%
- Answered without a robots.txt (44 x 404, 32 x 403, 7 other): 83/500, 16.6%
- No usable answer (server or gateway error, or no response): 146/500, 29.2%
  - Of these, never answered at all: 24/500, 4.8%
  - This bucket is dominated by CDN, DNS, and telemetry domains that serve no website (akamai.net, apple-dns.net, cloudfront.net, root-servers.net, ytimg.com). Tranco ranks DNS popularity, so they rank high.
- Among the 354 domains that answered like websites: 235/354 with a valid robots.txt, 66.4%

## Declared rules (all n/500, with share of the 235 valid files)

- Blocks all crawlers by default (Disallow: / under User-agent: *): 25/500, 5.0% (10.6% of valid)
  - 20 of the 25 default-deny files also name specific crawlers and allow them back in
  - Blockers include facebook.com, instagram.com, x.com, netflix.com, reddit.com, wsj.com, reuters.com, trustpilot.com
- Sets a crawl-delay: 31/500, 6.2% (13.2% of valid). Median 4 seconds, range 0 to 20
- Lists a sitemap: 162/500, 32.4% (68.9% of valid)
- Names at least one of the 7 AI crawlers: 81/500, 16.2% (34.5% of valid)

## AI crawlers (named by / fully blocked / block rate when named / share of 500 blocking)

- Bytespider (ByteDance): 49 named, 39 blocked, 80%, 7.8%
- CCBot (Common Crawl): 50 named, 38 blocked, 76%, 7.6%
- GPTBot (OpenAI): 60 named, 37 blocked, 62%, 7.4%
- ClaudeBot (Anthropic): 56 named, 35 blocked, 62%, 7.0%
- Google-Extended (Google AI training): 57 named, 34 blocked, 60%, 6.8%
- Meta-ExternalAgent (Meta): 37 named, 30 blocked, 81%, 6.0%
- PerplexityBot (Perplexity): 53 named, 28 blocked, 53%, 5.6%

Top 3 most-blocked outright: Bytespider (39), CCBot (38), GPTBot (37).
Most named: GPTBot (60). Highest block rate when named: Meta-ExternalAgent (81%), Bytespider (80%).
Aggregate: 362 naming events across the 7 crawlers, 241 are full blocks (66.6%).

## Quotable findings

1. Two thirds of the reachable top websites publish a valid robots.txt (235 of 354), and two thirds of those use it to hand crawlers a sitemap.
2. Only 1 in 20 of the top 500 domains blocks all crawlers by default, and 20 of those 25 immediately allow a hand-picked guest list back in.
3. The least permissive robots.txt files belong to the biggest platforms: Facebook, Instagram, X, Netflix, Reddit.
4. When a top site names an AI crawler in robots.txt, it fully blocks it two times out of three (241 of 362 naming events).
5. GPTBot is the most named AI crawler (60 sites), Bytespider the most blocked outright (39 sites), and roughly one top-500 domain in six now writes AI crawler rules by name (81 of 500).
