# Fact sheet: two doors into the same 500 Hacker News stories

Capture 2026-07-30. Python 3.14.3. Source of truth: `python run.py` on the shipped capture in `data/`. One page at a time, no concurrency, honest User-Agent, pauses between requests as recorded per row in `data/manifest.csv`.

## Headline table (500 stories per door)

| Metric | HTML door | JSON door | Ratio |
|---|---|---|---|
| Requests | 17 | 5 | 3.4x |
| Wire bytes total | 114,583 | 436,485 | JSON 3.8x heavier |
| Wire bytes per story | 229 | 873 | JSON 3.8x heavier |
| Decoded bytes per story | 1,473 | 873 | JSON 1.7x lighter |
| Median response time | 176 ms | 440 ms | HTML 2.5x faster |
| Polite wall clock | 963.6 s (16.1 min) | 4.3 s | JSON 223x faster |
| Extraction code | 28 lines | 13 lines | 2.2x |
| Coupling points | 12 | 8 | 1.5x |

- Compression: the HTML door serves gzip (a 43 KB page averages 6.7 KB on the wire). The JSON door ignores `Accept-Encoding: gzip` and sends its payload uncompressed, padded with an `_highlightResult` block per hit.
- Matched control (JSON at 30 items per request, equal request count to HTML): 17 requests, 448,146 B wire, median 328 ms. Fewer-but-bigger requests is not what makes JSON heavy; the payload shape is.

## The rate limit finding

- Hacker News robots.txt (read 2026-07-30): `User-Agent: *`, `Crawl-delay: 30`, `/newest` not disallowed.
- Walking `/newest` at exactly that published 30 s pace: **HTTP 429, body `Sorry.`, on the 5th page**, after 120 of 500 stories. Shipped probe: `data/rate-probe.csv`, produced by `python run.py --rate-probe --html-delay 30`. The initial capture attempt at the same pace was refused on the 6th page after 150; the edge of the budget wobbles between walks, the refusal does not.
- Not an address ban: the same URL served HTTP 200 from the same exit address minutes later. It is a short burst budget.
- At 60 s per page (double the published delay), the walk completed 17/17 pages, 510 stories, no refusal. The shipped capture is that run.
- The Algolia JSON door documents 10,000 requests/hour and the 5-request walk never came near it.

## Completeness and agreement

- Field fill, 500 records per door: every field 100% on both doors except `url` on JSON at 97.2%. The 14 gaps are self posts (Ask HN and similar) which have no external link; JSON reports null, the HTML door shows an internal `item?id=` link for the same fact.
- Cross-door agreement: 498 stories returned by both doors; 2 only-HTML and 2 only-JSON (the listing moved during the 16-minute HTML walk). Of the 498 shared: title 100%, url 100%, author 100% identical; points 99.0% and comment counts 99.6% identical, the differences being live counters that moved between the two reads, minutes apart.

## Mutation test (fragility, measured)

Outcomes: `survived` = every record, every field. `crash`/`empty` = loud failures a monitor sees. `partial` = records keep flowing with the affected field silently empty.

| Change class | HTML parser | JSON parser |
|---|---|---|
| Restyle (6 mutations: renamed classes, moved attribute, renamed wrapper) | survived 1/6 (3 empty, 2 partial) | survived 6/6 |
| Data-model change (4 mutations: renamed points/author/comments/id) | survived 2/4 (2 partial) | survived 0/4 (4 partial) |

- Of the 21 CSS class names in the page, exactly 1 appears anywhere in the JSON payload (`title`, which is also a JSON field name, by coincidence of vocabulary). A restyle has essentially nothing to reach in the JSON.
- Every JSON failure under a data-model change was `partial`: the parser kept returning records with the renamed field empty. The API door does not break less under contract changes here; it breaks more quietly.

## Hypothesis verdict (brief, written before the run)

1. "The JSON door is several times faster": **wrong per request** (HTML answered 2.5x faster: 176 vs 440 ms median), **right operationally** for a different reason: at the paces the sources tolerate, 4.3 s vs 16.1 min, 223x, driven by the rate limit and page size, not server speed.
2. "An order of magnitude lighter": **wrong on the wire**, inverted: JSON cost 3.8x more bytes because the API refuses compression and pads every hit. Only after decompression is JSON lighter, 1.7x, nowhere near an order of magnitude.
3. "Cannot break from a layout redesign": **confirmed** (6/6 vs 1/6), with the brief's own caveat sharpened by data: under data-model changes the JSON parser failed 4/4 and every failure was silent.

## Quotable findings

1. Collecting 500 stories politely took 16.1 minutes through the HTML door and 4.3 seconds through the JSON door. The 223x is rate limits and page size, not server speed: per request, the HTML door answered 2.5x faster.
2. Hacker News publishes a 30-second crawl delay and refused a crawler honoring exactly that pace on the fifth page. At 60 seconds it served all 17. The published rate and the enforced rate are two different numbers.
3. The JSON door billed 3.8x more bytes on the wire than the HTML door for the same stories, because the API sends uncompressed, padded payloads while the site gzips its pages. "JSON is lighter" is a claim about your parser's input, not your bandwidth bill.
4. A six-mutation restyle broke the HTML parser five ways and the JSON parser zero; of 21 class names in the page, 1 appears in the JSON. But all four data-model mutations put the JSON parser into partial mode: rows kept flowing with a field silently empty.
