# Two doors into the same 500 records

Capture date: 2026-07-30. Source: Hacker News, which publishes the same listing of new stories through two doors. Python 3.14.3. All numbers below come from the capture shipped in `data/`, reproduced by `python run.py`.

## The question

Chapter 3 claims that the site's own JSON feed is usually a better door than its rendered HTML. That is a claim about cost, fragility, and completeness, so it should be measured rather than asserted.

Hacker News is a fair place to measure it, because the same content is genuinely available both ways:

- **The HTML door.** `news.ycombinator.com/newest` renders 30 stories per page and paginates through a "More" link.
- **The JSON door.** `hn.algolia.com/api/v1/search_by_date?tags=story` returns the same newest-first listing as structured data, up to 100 stories per request.

Both are list-shaped, so this is a door-versus-door comparison rather than a rigged one. Hacker News also publishes an item-by-item API, which would need one request per story and would have lost this comparison on request count before it began. It is deliberately not used.

The target: collect the newest 500 stories through each door, extract the same seven fields from both (`id`, `title`, `url`, `points`, `author`, `num_comments`, `created_at`), and measure what each door charged for them.

## What is measured

| Metric | Why it is in the list |
|---|---|
| Requests to collect 500 stories | Requests, not bytes, are what rate limits count |
| Bytes as they crossed the wire | What bandwidth and per-gigabyte proxy billing actually charge for |
| Bytes after decompression | What the parser sees, which is a different number |
| Server response time | The door's mechanical speed, isolated from politeness |
| Wall clock at a tolerated pace | The operational reality: how long the job really takes |
| Lines of extraction code | A proxy for how much there is to write and maintain |
| Coupling points | How many things about the response the parser depends on |
| Field fill rate and cross-door agreement | Completeness, and whether the doors tell the same story |
| Mutation survival | Fragility, measured instead of assumed |

## Method

Both parsers live in `run.py` between explicit `PARSER` markers and produce identical record shapes, so the code-size comparison is like for like. Neither is golfed and neither hides work in a shared helper.

The capture is polite by construction. One page at a time, no concurrency, an honest identifying User-Agent, and a pause between requests. Chapter 2's rule applies here: where a site declares a crawl-delay, that number wins. Hacker News declares `Crawl-delay: 30` for all crawlers in its robots.txt, and `/newest` is not among its disallowed paths.

That declared number turned out to be the most interesting thing in the experiment, so it gets its own section below.

### The matched control

The JSON door returns 100 stories per request and the HTML door returns 30. That difference is real and it is part of why the JSON door wins on request count, but it would confound any per-request timing comparison. So the capture also collects the JSON door a second time at 30 stories per request, which holds request count equal to the HTML door and isolates "fewer requests" from "faster per request". It is recorded as the `json_matched` door in `data/manifest.csv`.

### The mutation test

Fragility is usually asserted. Here it is tested by breaking both parsers on purpose, in two classes of change:

- **A restyle**, six changes: renaming CSS classes, moving an attribute, renaming a wrapper. These are the changes a visual redesign makes, and none of them alters what the data means.
- **A data-model change**, four changes: renaming the points field, the author field, the comment count, and the identifier. These are the changes an API deprecation or a schema revision makes.

Each mutation is applied to a captured page and the parser is re-run. The outcome is recorded in one of four buckets, and the difference between the last two is the point of the exercise:

- `survived`: every record, every field.
- `crash`: an exception. Loud, and a monitor will see it.
- `empty`: zero records returned. Also loud.
- `partial`: records still come back, with the affected field quietly empty. This is the dangerous one, because the pipeline stays green while the data rots.

One limit stated plainly: a restyle is applied only to the HTML door, because a restyle cannot reach the JSON payload. That is not a measurement, it is where the data sits. The check on that claim is a count of how many of the page's CSS class names appear anywhere in the JSON response.

## What the run showed

### The pace was the first result

The capture plan honored the crawl-delay Hacker News publishes, 30 seconds. The site refused it. On the initial attempt the sixth page answered HTTP 429 with the body `Sorry.`, after 150 stories; a fresh probe at the same pace was refused on the fifth page, after 120. The refusal is not an address ban, because the same URL served normally from the same exit address after a rest; it is a short burst budget. At 60 seconds between pages, double the published delay, all 17 pages were served without complaint. The shipped capture is that 60 second walk, and `data/rate-probe.csv` holds the probe.

Two practical readings. First, the published rate and the enforced rate are different numbers, and only the source knows the second one. Second, a collector must treat 429 as a signal to slow down, not retry, which is Chapter 6's back-off discipline arriving one chapter early.

### Headline numbers (500 stories per door)

| Metric | HTML door | JSON door | Ratio |
|---|---|---|---|
| Requests | 17 | 5 | 3.4x |
| Wire bytes per story | 229 | 873 | JSON 3.8x heavier |
| Decoded bytes per story | 1,473 | 873 | JSON 1.7x lighter |
| Median response time | 176 ms | 440 ms | HTML 2.5x faster |
| Wall clock at tolerated pace | 963.6 s | 4.3 s | JSON 223x faster |
| Extraction code | 28 lines | 13 lines | 2.2x |
| Coupling points | 12 | 8 | 1.5x |

The wall clock row decides the operational question, and it is driven by the rate limit and the page size, not by server speed: per request the HTML door was the faster one. The wire bytes row is the surprise. Hacker News gzips its pages, about 43 KB of markup crossing as 6.7 KB, while the search API ignores `Accept-Encoding: gzip` and pads every hit with an `_highlightResult` block, so the "lightweight" JSON door billed 3.8x more bandwidth for the same stories. The matched control rules out request granularity as the cause: at 30 items per request the JSON door still moved 448,146 bytes over 17 requests, essentially the same per-story weight.

### Completeness and agreement

Every field arrived 100% through both doors except `url` on the JSON side, 97.2%: the 14 self posts (Ask HN and similar) have no external link, which JSON reports as null and the HTML door papers over with an internal `item?id=` link. Of the 498 stories both doors returned (2 more appeared in each door's window as the listing moved during the 16 minute HTML walk), titles, links and authors matched at 100%; points matched at 99.0% and comment counts at 99.6%, the gaps being live counters that moved between the two reads, minutes apart. Agreement should be judged on stable fields; counters measure the clock as much as the door.

### Mutation results

| Change class | HTML parser | JSON parser |
|---|---|---|
| Restyle, 6 mutations | survived 1/6 (3 empty, 2 partial) | survived 6/6 |
| Data-model change, 4 mutations | survived 2/4 (2 partial) | survived 0/4 (4 partial) |

The restyle row is as lopsided as the chapter's claim needs it to be, and the CSS token count explains why: of the 21 class names in the page, exactly one string (`title`) coincides with anything in the JSON payload, and that by accident of vocabulary. The data-model row cuts the other way and is the finding worth remembering: every JSON failure was `partial`, records flowing with the renamed field silently empty. The API door breaks less often and more quietly. Neither door exempts you from validating what arrived.

### The hypothesis, judged

The brief predicted the JSON door several times faster, an order of magnitude lighter, and immune to layout redesigns. The run confirmed the immunity (6/6 against 1/6), inverted the per-request speed (HTML answered 2.5x faster; JSON wins wall clock 223x for rate-limit and page-size reasons), and inverted the weight on the wire (JSON 3.8x heavier; only 1.7x lighter after decompression). Two of three mechanisms wrong, conclusion still standing, which is exactly why the run happens before the prose.

## Reproducing this

```
pip install -r requirements.txt
python run.py
```

The shipped capture makes the run fully offline, so the numbers above are the numbers you get. `python run.py --refresh` re-collects both doors live at a pace the source tolerates, and `python run.py --rate-probe` reproduces the rate-limit finding at the published crawl-delay. Live runs will differ from the printed numbers, because the listing moves.
