# What the top 500 websites actually say in robots.txt

Snapshot date: 2026-07-29. Sample: the top 500 domains of the Tranco research ranking, list ZJL5G (https://tranco-list.eu/list/ZJL5G). Python 3.14.3. All numbers below come from the shipped snapshot in `data/`, produced by `python run.py`.

## The question

The most visited sites on the web publish a small file called robots.txt. What do those files actually say? How many sites block every crawler, how many ask for a slower pace, and which AI crawlers get singled out by name?

## Method, in plain words

A robots.txt is a plain text file at a fixed address (`example.com/robots.txt`) where a site states which automated visitors, called crawlers, may fetch which pages. It is a stated preference, not a lock: nothing in it can physically stop a request. That is exactly why it matters for this chapter. Respecting it is the clearest signal that a collector operates in good faith.

The census works like this:

1. Take the top 500 domains from Tranco, a research ranking of popular domains (list ZJL5G, so anyone can pull the exact same sample).
2. Fetch `https://<domain>/robots.txt` once per domain, with an honest identifying User-Agent, a 10 second timeout, at most one retry, at most 10 domains in flight, and a single plain-http fallback when https fails at the network level. Failures are recorded in `data/manifest.csv` as results, not retried into submission. The whole run moved 8.7 MB, less than a single news homepage.
3. Save the 271 bodies that came back with HTTP 200 into `data/robots-snapshot-2026-07-29/`, one file per domain.
4. Parse each file with protego, a strict robots.txt parser, plus direct line reads for the counting of named crawlers. A file only counts as a valid robots.txt if it returned HTTP 200 and contains at least one real robots directive. HTML error pages served with status 200 do not count.

The book's numbers come from the shipped snapshot, so your run of `python run.py` reproduces this census exactly, offline; the finer file-level details quoted in the chapter read straight out of `data/manifest.csv` and the snapshot folder. `python run.py --refresh` re-fetches a fresh snapshot with the same polite rules, and your numbers will then differ slightly, because the web moved.

## What one fetch per domain came back with

| Result | Count | Share of 500 |
|---|---|---|
| A valid robots.txt | 235 | 47.0% |
| HTTP 200, but the body is not a robots.txt | 36 | 7.2% |
| Answered without one (mostly 404 and 403) | 83 | 16.6% |
| No usable answer (server error, or nothing) | 146 | 29.2% |

The last row looks alarming until you look at who is in it: `akamai.net`, `apple-dns.net`, `cloudfront.net`, `root-servers.net`, `ytimg.com`. Tranco ranks domains by how often the world's machines look them up, so its top 500 is full of content-delivery, DNS, and telemetry hosts that serve software, not people. A dead connection there is the expected answer. Counting only the 354 domains that answered like websites, 235, or 66.4%, publish a valid robots.txt. Two thirds of the reachable web's front row states its crawling preferences in a machine-readable file.

## The census

Shares of all 500 domains, with counts:

- **Has a valid robots.txt: 235 (47.0%).** Among reachable websites, 66.4%.
- **Lists a sitemap: 162 (32.4%).** A sitemap line points crawlers at a directory of the site's pages. 162 of the 235 valid files carry one (68.9%). Sites overwhelmingly use robots.txt to invite orderly crawling, not to forbid it.
- **Names at least one AI crawler: 81 (16.2%).** One third of all valid files (34.5%) single out at least one of the seven AI crawlers counted here.
- **Sets a crawl-delay: 31 (6.2%).** Crawl-delay asks a crawler to pause between requests. Declared values run from 0 to 20 seconds with a median of 4. Examples: `github.com` asks for 1 second, `wikipedia.org` for 5, `europa.eu` for 10.
- **Blocks all crawlers by default: 25 (5.0%).** These files put `Disallow: /` under `User-agent: *`, which refuses every page to every crawler that is not named separately. That is 10.6% of the valid files, and the names are striking: `facebook.com`, `instagram.com`, `x.com`, `netflix.com`, `reddit.com`, `wsj.com`, `reuters.com`, `trustpilot.com`. Facebook's default rule is two lines long: `User-agent: *`, `Disallow: /`. But 20 of these 25 then name specific crawlers and let them back in one by one. Default-deny plus a guest list is the big-platform pattern, not a closed door.

## The AI crawler picture

For each crawler: how many of the 500 sites name it in robots.txt, and how many of those namers disallow everything for it.

| Crawler | Operator | Named by | Fully blocked | Block rate when named | Share of all 500 |
|---|---|---|---|---|---|
| Bytespider | ByteDance | 49 | 39 | 80% | 7.8% |
| CCBot | Common Crawl | 50 | 38 | 76% | 7.6% |
| GPTBot | OpenAI | 60 | 37 | 62% | 7.4% |
| ClaudeBot | Anthropic | 56 | 35 | 62% | 7.0% |
| Google-Extended | Google, AI training | 57 | 34 | 60% | 6.8% |
| Meta-ExternalAgent | Meta | 37 | 30 | 81% | 6.0% |
| PerplexityBot | Perplexity | 53 | 28 | 53% | 5.6% |

Across the seven crawlers there were 362 naming events, and 241 of them are full blocks. When a top site bothers to name an AI crawler, it blocks it two times out of three. GPTBot is the most named (60 sites). Bytespider is the most blocked outright (39 sites), with CCBot (38) and GPTBot (37) just behind. The blockers include `nytimes.com`, `amazon.com`, `cnn.com`, `ebay.com`, and `yahoo.com`.

## Verdict on the hypothesis

The brief predicted that blanket blocking of all crawlers would be rare, that most sites would express preferences rather than walls, and that AI-crawler blocking might be far more common than industry folklore expects. The snapshot supports all three, with one twist.

Blanket blocking is a minority position, 25 sites of the 235 with a valid file. Most files are invitations with conditions: two thirds list a sitemap, a slice asks for a slower pace, and even the default-deny platforms mostly maintain a guest list rather than a wall. The twist is who the blockers are: the least permissive robots.txt files on the web belong to the biggest consumer platforms.

The AI wave is real and measurable. A third of valid files now name at least one AI crawler, and naming is usually a prelude to blocking. For a data collection professional the practical reading is simple: the industry norm is written down, it is specific, and it increasingly says that AI training crawlers are not welcome where general crawlers still are. Reading the file before you fetch is not politeness theater. It is how you find out what the site already told you.

## Reproducing this

```
pip install -r requirements.txt
python run.py            # parses the shipped 2026-07-29 snapshot, prints the census, writes the charts
python run.py --refresh  # optional: re-fetch a fresh snapshot politely, then run the same census
```

Python 3.14.3, dependencies pinned in `requirements.txt`. The charts land in `../images/` as `robots-census` and `ai-crawler-blocks`, in SVG and PNG.
