The Data Collection Handbook · Part II. Getting the Data

Chapter 7. Extraction: From Page to Record

David Martin Riveros · Published August 4, 2026

The bytes are not the data. This is the sentence I wish someone had said to me early, because for a long time I thought the hard part of collection was getting the page, and that once the page was in hand the record fell out of it. It does not. Between the response you fetched and the row your customer reads sits a step where you decide, field by field, what the page means. That step is extraction, and it is where data is actually born. A fetch that works and an extractor that is subtly wrong do not cancel out. They combine into a confident, well-delivered, wrong number.

Chapters 3 through 6 were about getting bytes off the internet without lying and without breaking. This chapter is the step everyone skips in the telling, the one between the bytes and the validated row: turning a page into a record. It has one habit that saves more grief than any other, one place to look before you write a single selector, and one failure mode that is the quiet cousin of Chapter 6's forbidden edge. Get those three right and extraction stops being the part that breaks every Tuesday.

Store the raw response, always

Here is the single best habit in the whole discipline, and it costs almost nothing: store the raw response before you parse it. Fetch the page, write the exact bytes to storage, and then parse from the stored copy, not from the wire. It sounds like bookkeeping. It is the difference between a parsing bug that costs you an hour and one that costs you the data.

Think about what a parsing bug is. Your extractor has been reading a field wrong, maybe for a week, maybe since the day it shipped. You discover it. Now, what do you have to work with? If you parsed straight from the wire and threw the bytes away, the answer is nothing: to fix the past you would have to re-fetch every page, which means paying for the requests again, putting that load on the source again, and hoping the pages even still say what they said last week, which for prices and stock they do not. The history is simply gone. But if you stored the raw responses, the fix is a re-parse. Correct the extractor, run it back over the archive you already have, and the past repairs itself with no new request, no new cost, and no new load on anyone.

Storage is the cheapest thing in this entire book. A compressed HTML page is a few kilobytes; a year of a large collection is a rounding error against what the collection itself costs to run. What that trivial spend buys is enormous: every parsing fix applies retroactively, you can audit exactly what the source served on any past date, and the morning a site redesigns you can diff today's raw response against yesterday's and see precisely what moved. I have debugged a customer's angry question about a number from three weeks ago by opening the stored response from that day and reading it with them on the call. Without the archive that conversation is a shrug. With it, it is a fact. Treat storing the raw as non-negotiable, the way Chapter 2's checklist treats never fetching what you already have; they are the same discipline seen from two sides.

The data is usually already in the page

Now the place to look before writing any selector, and it is the habit that separates people who have done this for years from people who are about to have a hard month. A modern web page is not really a document. It is a program that a site hands your browser, and that program almost always arrives carrying its data as structured JSON, right there in the page, before a single pixel is drawn. Chapter 3 met this as the hidden JSON door. Here it comes back one level in: even when you are working with the rendered HTML, the clean structured data is usually embedded inside that HTML, and you should reach for it first.

There are three common forms, and it is worth knowing their shapes by sight. The first is the framework state blob: modern sites built on common frontend frameworks embed the whole page's data in a script tag, often under a name like the one this chapter's example uses, so the code that draws the page has something to draw from. The second is JSON-LD, a block of standardized product data that sites publish for search engines, sitting in a script tag that announces itself as structured data. The third is an inline API response, the same JSON the site's own frontend fetched, sometimes baked directly into the page. All three hand you names, prices, and identifiers as actual typed fields, not as text you have to dig out of visual markup.

The reason this matters is not tidiness. It is survival. The visual HTML, the divs and spans a designer arranges, is the single most redesigned thing on the internet; it changes shape every time someone tweaks the layout. The embedded data structures change far less often, because they are the site's own plumbing and breaking them breaks the site's own frontend. When you read a price out of a JSON blob instead of out of a styled span, a visual redesign sails right past your extractor, because you were never reading the visuals in the first place. You were reading the same source the visuals themselves read from.

The extraction ladder

Put those choices in order and you get a ladder, most durable at the top, and the rule is simple: climb to the highest rung the page actually offers.

The top rung is the embedded JSON just described: the site's own data, the most redesign-proof source there is. Below it are stable semantic attributes, the hooks a site deliberately puts in its markup to be stable, things like a test identifier or a standardized data attribute; they exist precisely so that code can grab an element without caring how it looks, which is exactly what you want. Below that are resilient selectors: when you must read the visuals, anchor on meaning rather than position, find the price by its relationship to a label that says Price, not by its coordinates in the tree. And at the bottom is the brittle positional path, reaching for the third div, the second span, a field by where it happens to sit. That bottom rung works on the day you write it and is pure debt from then on, and the experiment in this chapter is about what that debt costs when it comes due.

The experiment: how extractors fail, not just whether

I have argued that the top of the ladder survives redesigns and the bottom does not. But survival was never the interesting question. The interesting question is how each one fails, because failures come in two flavors that are worlds apart, and I wanted to watch it happen rather than assert it.

So the example ships the same product page across five eras: a baseline, then four redesigns modeled on the ones I see in the wild, a class rename, a layout swap, a framework migration, and a change to the shape of the embedded data itself. One fictional product, one true set of values, the same page dressed five ways. Three extractors are written honestly against the baseline, the way a developer would on day one, and then run unchanged against all five eras. Each result is scored against the known truth as one of three things: correct, a loud failure, meaning it returned nothing and raised an error you would see, or a silent wrong, meaning it returned a complete, ordinary-looking record with a value that is simply false. Everything is offline; python run.py reproduces the whole matrix.

The tallies came out about as the ladder predicts, and I will give them plainly. The embedded-JSON extractor got four of the five eras right and survived the most. The semantic-attribute extractor got three. The brittle positional path got one, the baseline it was born on, and broke on everything after. If the chapter ended there it would be a tidy argument for reading the JSON, and not a very surprising one.

But look at the one red cell, because it is the entire point. On the layout-swap era, the brittle extractor did not crash. The redesign had put a struck-through original price into the markup ahead of the current one, both now plain price spans, and the positional extractor, told to grab the price span, grabbed the first one it found. It returned a perfect, complete, confident record for the Trailhead bottle at 32.00 dollars. The real price was 24.50. Nothing errored. Nothing looked wrong. That record would have been delivered, billed, and used to make a pricing decision against a competitor's number that was never real. It is Chapter 6's forbidden edge wearing a different coat: not a failure recorded as a value, but a wrong value recorded as a success, and it is worse than a crash for exactly the same reason. A crash announces itself and gets fixed by lunch. A silent wrong hides inside a valid-looking feed and surfaces, if it ever does, as someone downstream asking why a decision went bad.

Now the part that keeps the chapter honest, and it corrected a lazy version of this argument I have made before. The embedded-JSON extractor was not magic, and it did not survive everything. On the fifth era, where the shape of the embedded data itself changed, it failed too. But how it failed is the whole lesson: it failed loudly. Because it was written to validate the shape it expected, a price where a price should be, it found the schema it knew was gone and raised, returning no record rather than a wrong one. Zero silent failures across all five eras, for both the JSON and the semantic extractors. Only the brittle path lied. So the honest conclusion is not read the JSON and you are safe. It is that the durable strategies fail in the safe direction, loudly, where you will see them, and the brittle one fails in the dangerous direction, quietly, where you will not. That is the property you are actually buying when you climb the ladder.

Normalize on the way in

One more job belongs to extraction, done at the moment a field enters, and skipping it pushes a mess downstream that is much harder to clean once it has spread. The same fact arrives in many costumes: a price as one thousand two hundred ninety nine and change might come as a European-formatted string with the comma and dot swapped, a currency as a symbol here and a code there, a date in three regional orderings, text wrapped in stray whitespace or odd encodings. Normalize each one as it comes in: a price becomes a number plus an explicit currency code, a date becomes one standard format, whitespace and encoding get cleaned at the gate. The rule of thumb I hold to is that everything past the extractor should see one consistent shape, so that no consumer downstream has to guess whether this row's price is in the same units as that row's. A field you normalize once, at the door, is a field nobody has to reinterpret ever again.

Failing loudly is the whole point

Which brings the chapter back to its spine, the thread that ties it to Chapter 6 and forward to the monitors of Chapter 10. When an extractor cannot find what it came for, it must fail loudly. Not return an empty record, not fill the field with a zero or a blank, not quietly move on. It must raise, so the miss lands in Chapter 6's third bucket as an ambiguous failure and becomes a retry or an honest gap, never a value. The experiment made the stakes concrete: the extractors that raised on a schema they did not recognize protected the dataset, while the one that dutifully returned whatever it found poisoned it. An empty record that pretends to be full is the same lie as a block recorded as a price, born one step later in the pipeline.

That is the mindset extraction demands and the reason it deserves its own chapter. The fetch got you bytes. The extractor decides what those bytes mean, and every field a customer sees is one of its decisions. Store the raw so a wrong decision is reversible. Read the site's own JSON so fewer decisions depend on how the page looks. And wire every parse miss to fail loudly, so the pipeline can never mistake a guess for a fact. With the record honestly born, the next question is whether it is any good, which means checking it, at scale, after the fact. That is Chapter 8.