The Data Collection Handbook · Part II. Getting the Data
Chapter 7. Extraction: From Page to Record
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Fetching got you bytes. Every field your customer ever sees is a parsing decision made after that. This chapter treats extraction as its own discipline, starting with the single best habit in the industry: keep the raw response, so any parsing bug can be fixed by re-parsing your archive instead of re-fetching the web.
What you will learn
- The archive-first pattern: fetch, store the raw response, parse from the store. Storage is cheap; re-fetching history is impossible.
- How to find the structured data most modern pages already carry inside them, before writing a single visual selector.
- The extraction ladder: embedded data first, then meaningful attributes, then resilient selectors, with brittle positional paths as marked debt.
- Normalization at the gate: units, currencies, dates, and encodings, so downstream consumers see one consistent shape.
- Why a parse that finds nothing must raise an error, never return an empty record.
In this chapter
- The parse is where data is born. A wrong selector is a factory for wrong data.
- Store the raw, always. What the archive costs and what it buys.
- The treasure already in the page. Finding the site's own data structures inside its pages.
- The extraction ladder. Four rungs ranked by survival odds.
- Selectors that survive. Anchoring on meaning instead of position.
- Normalization at the gate. One price field's journey to a clean number.
- Failing loudly. Wiring parse failures into the three buckets of Chapter 6.
The experiment
The question: Across several redesigns of the same page, which extraction strategy keeps working, and when strategies fail, which fail loudly versus silently shipping wrong data?
Shipped snapshots of one product page across several redesign eras, modeled on real-world patterns: class renames, layout swaps, and a framework migration. Three extractors are written against the first era only, then run against every era, scoring each result as correct, loud failure, or silent wrong data. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that the embedded-data extractor survives the most redesigns, and that the brittle positional path not only breaks first but produces at least one silent wrong-data result, which is far worse than any crash.