The Data Collection Handbook · Part V. The Other Side of the Table

Chapter 14. How to Buy Data Well

David Martin Riveros · Published August 23, 2026

I sell data for a living. For thirteen chapters that worked in your favor; on this last one it makes me the party under audit, because the chapter switches chairs and teaches you to buy. I am writing it anyway, mostly out of self-interest. A buyer who can check the work steers money toward vendors who do the work, and a vendor with nothing to hide would rather compete on an audit than on a claim. The ones who lose when buyers learn these checks were never selling data in the first place.

Start with the shape of the disadvantage. Your vendor watched the collection run: which pages blocked, which categories the crawler never reached, which rows came from an old cache, what broke midweek and how it was patched. You see none of it. You receive a file, an invoice, and a claim, and the claim is usually a round, confident number. Economists call this information asymmetry, one side knowing more than the other about the goods. George Akerlof's Nobel-winning example was used cars, and the used-car world's working answer, the independent inspection before purchase, is what this chapter hands you.

The gap never fully closes, and it does not need to. What you need is this book's method held from the other end: checks you can run yourself, questions whose answers are hard to fake, the contract terms a measurement can trigger, and a rule for walking away. None of it needs code, and most fits in an afternoon.

An audit a spreadsheet can run

Ask every vendor on your shortlist for a sample delivery against your real specification; some will refuse, and that is worth knowing before any money moves. When the sample lands, run five checks. They are Chapter 8's validation discipline with the roles reversed, and each ends in one number against one limit, which later lets the same five lines go into a contract.

Fill. One COUNTBLANK per promised column, the spreadsheet function that counts empty cells; every required column must come back at least 98% filled. It catches broken extraction and forgotten fields, and it is the weakest of the five, because a fabricated row fills every cell beautifully. Run it first and trust it least.

Duplicates. Add three helper columns: the SKU (the product code) in upper case with spaces and punctuation stripped, the product name in lower case with punctuation stripped, the URL cut at the question mark. One COUNTIF per helper column, counting matches among the rows above, flags every collision. More than 2% of rows colliding fails. Honest pipelines leave a little residue; padded files leave a lot, because a disguised copy is the cheapest padding there is.

Freshness. Split the timestamp column and count rows whose time part reads exactly 00:00:00. More than 5% fails; the allowance covers schedulers that genuinely fire at midnight. A crawler stamps 14:05:09 because it ran at 14:05:09; a person re-dating a column types 2026-08-12 and the clock silently becomes midnight. While you are there, count the distinct days and set them against the claimed crawl window.

Impossible rows. Write down rules the real world enforces and flag rows that break one: a strikethrough price, the crossed-out was-price next to a discount, must sit above the selling price; a rating cannot rest on zero reviews; one SKU cannot hold two prices at the same moment; price and stock must hold legal values. More than 1% flagged fails. Real catalogs produce a few of these through source bugs, and padding produces them in bulk.

The coverage claim. Homework first: an afternoon on the public source, writing down 60 products by hand, spread across categories, popular and obscure. Look up each of the 60 in the delivery with VLOOKUP, the spreadsheet function that finds a value in another table. The hit rate is the vendor's coverage as measured by you, and a claim more than 10 points above it fails. Ten points, because a 60-item sample carries its own noise: at 95% confidence near an 80% true rate the sampling margin is about 10 points, so the tolerance forgives sampling luck and nothing more. The overlap also carries Chapter 9's capture-recapture trick, pointed the other way: the delivery's distinct product codes times 61, divided by your hits plus 1, estimates the size of the catalog the file was really drawn from.

Two vendors, one of them padded

Would this audit catch a real cheat? The experiment stages it. Both vendors are simulations, generated from a fixed seed (python run.py reproduces every number and both charts offline), with padding rates chosen to be plausible rather than measured from any real vendor. Simulation is the only fair option: with real deliveries I could never publish the answer key, and the answer key is what grades the audit itself.

The stage is a catalog of 1,200 products whose true size only the grader knows. Vendor A delivers 989 rows for $1,200 and claims 84% coverage. Its file is what a good real crawl looks like: misses clustered behind two categories' pagination walls, 5 leftover duplicate rows, 3 rows where the source page showed no brand, 2 with a mis-set strikethrough price, real blanks where products have no reviews or discount. Vendor B delivers 1,150 rows for $950 and claims 96%, "every row refreshed within 72 hours." Its file is built the way padded files really get built. A fresh core of 400 truly crawled rows. Then 260 stale rows re-dated to look fresh. Then 190 copies of rows already in the file, lightly disguised: a case flip on the SKU, an appended suffix, a tracking parameter on the URL, a price moved by one cent. Then 300 products fabricated from a template. Count rows and B is the fuller file at the better price.

The audit breaks the tie in four moves.

Fill flatters the cheat, as warned. Vendor B scores 100.0% on every required column, because fabricated rows never leave a blank. Vendor A's weakest required column is brand at 99.7%, those 3 pages that showed none. Both pass the 98% line. If the audit stopped here, and most stop here, the padded file wins.

The duplicate check ends the illusion. Vendor A: 5 of 989 rows collide on a normalized key, 0.5%. Vendor B: 190 of 1,150, 16.5%, eight times over the 2% limit, because every disguised copy collides with its source row on at least one helper column. A disguise that fools an eyeball does not survive an upper case and a strip.

Then the clock. Vendor A has 0 of 989 rows stamped at exact midnight, 0.0%, and its timestamps spread across 6 distinct days matching the crawl window it claimed. Vendor B has 260 rows at exactly 00:00:00, 22.6% against the 5% limit, bunched on just 2 of its 3 claimed days. Those are the re-dated stale rows. The prices they carry are months old, which no formula can see; the timestamps gave the re-dating away.

The impossible-row rules flag 139 of Vendor B's rows, 12.1% against a 1% limit: 47 rows with a strikethrough price at or below the selling price, 50 rows carrying a rating with zero reviews, and 49 SKUs holding two different prices at the same moment, that last group being the one-cent disguise contradicting its own source row. Seven fabricated rows break two rules at once, which is why the three counts add past the total. Vendor A has 2 flagged rows, 0.2%, its two mis-set strikethroughs, the small residue the 1% allowance exists for.

Before any of this ran, the brief committed to a hypothesis: the padded vendor fails at least 3 of the 5 checks on internal evidence alone, no ground truth needed. Judged against the run: confirmed, right at its bar. The 3 failures that use nothing but the delivered file itself, duplicates, freshness, and impossible rows, are the at-least-3 the hypothesis named, and the spot list adds a fourth from outside. Vendor A passes all 5, which matters just as much, because an audit that condemns everyone teaches nothing.

Your 60 rows against their 96%

The fifth check catches the one trick the other four cannot.

Look up the 60 hand-gathered products in each delivery. Vendor A: 50 of 60 found, 83.3% measured against an 84% claim, a gap of 0.7 points. Vendor B: 38 of 60 found, 63.3% measured against a 96% claim, a gap of 32.7 points, three times the tolerance. The grader's answer key, which the audit never sees, says the little list read both fairly: Vendor A truly covers 984 of the 1,200 products, 82.0%; Vendor B, 660 of them, 55.0%.

One more calculation from the overlap. Vendor A's overlap implies a catalog of about 1,177 items, close to the true 1,200. Vendor B's 1,021 distinct product codes against its 38 hits imply about 1,597. (Why 1,021 codes when check 2 left only 960 fully unique rows: 61 of the disguised copies changed the product code while still colliding on name or URL. Run the estimate on either base and the manifest is contradicted.) Hold that against B's own paperwork: its manifest states a catalog of 1,200 items, its claim needs 96% of that, and its file behaves like a sample of a universe around 1,600 items. The padding failed to support the claim and then went further, manufacturing a second, contradictory claim inside the same delivery.

There is a five-minute version of this check that needs no hand list at all. Count the rows that survive the duplicate check, then divide by the catalog size the vendor's own manifest states, or better, by the count the public source displays on its own category pages. That number is a hard ceiling on any coverage claim. Vendor B has 960 unique rows against its own stated catalog of 1,200, a ceiling of 80%, so its 96% claim dies before the spot list is even opened. Vendor A's 984 unique rows against its stated 1,170 read 84%, matching its claim. I run this division on every quote before I spend the afternoon.

The real reason the spot list exists: of Vendor B's 300 fabricated rows, 213 survive every row-level check in this audit, because a template can be internally consistent, prices above strikethroughs, ratings resting on plausible review counts, every cell filled. Read one at a time, they are unimpeachable. What they can never do is show up on a list you gathered from the real catalog, so in aggregate they dilute the hit rate and inflate the implied catalog size, and the cross-check converts their existence into evidence. Fabricated rows hide one by one and surface all at once.

Two limits. The midnight test catches lazy re-dating; a careful padder randomizes fake times and beats check 3, so the verdict rests on the two things hardest to fake, the duplicate structure and the coverage arithmetic. Against careful re-dating the leftover defense is manual: open a dozen delivered rows on the live site and compare prices. And a fraud built to pass all five would have to actually crawl the catalog fresh and wide, at which point it has become the product it was imitating, which was the point all along. The audit cannot make cheating impossible, but it can make cheating cost more than honesty.

The price per row you actually pay

Sticker prices said Vendor B was the bargain: $950 for 1,150 rows is $0.83 per row, against Vendor A's $1,200 for 989 rows, $1.21. Drop every row the audit flagged and the order flips. Vendor A keeps 982 of its 989 rows, so its usable rows cost $1.22 each, one cent over sticker. Vendor B keeps 650 of 1,150, and its usable rows cost $1.46 each. The quote that looked 32% cheaper per row now runs 24 cents per row more than Vendor A's.

The grader sees one level further than any real buyer could. 213 of B's 650 surviving rows are fabricated, so the rows that are both usable and real cost $950 for 437 rows, $2.17 each. Chapter 11 crowned cost per successfully delivered record the master metric of a collection operation. This is its buyer-side twin, price per genuine usable row, and the sample audit gives you a first estimate before anything is signed.

Seven questions, and how to read the answers

So far you have tested a file. These questions test the operation behind it, each mapping back into this book: you are checking whether the vendor runs the operation these chapters described, without seeing it. Put all seven in writing, and read the answers for numbers and documents you could ask to see.

1. How do you know when your collection breaks? You want monitoring built on the data itself, fill rates that sag and distributions that shift, with alarms that learn each field's normal wobble (Chapter 10). "Our customers tell us" means the monitoring is you.
2. Show me last month's quality report for a customer like me. An operator produces a redacted real one, because one already exists for every delivery; redacting it costs a real vendor an hour, and I know because I have spent that hour. If yours would be the first ever written, the product you are quoting does not exist yet.
3. How do you bill failures? Chapter 6 drew its forbidden edge here: never fabricate a result from a failure. The billing version: does a blocked page cost me money, and how would a made-up success be caught before my invoice? A vendor who charges per attempt has moved collection risk onto your budget. This is the question I would brace for in the seller's chair, because per-attempt billing is comfortable to run and uncomfortable to defend out loud.
4. What happens to my delivery when a source redesigns? A real answer names mechanisms and a window: stored raw responses that re-parse without re-fetching (Chapter 7), and a declared gap while the extractor is rebuilt. "It rarely happens" means there is no plan.
5. How do I verify a delivery landed complete? You want a manifest with every delivery and verification at the destination storage rather than a status flag, Chapter 12 played from your side of the net.
6. When something goes wrong, what exactly gets re-delivered, and who notices first? Re-delivery must be safe to repeat without doubling rows, and "who notices first" should be them, through Chapter 13's incident discipline.
7. What exactly may I do with this data? The license question, and the next section's subject. Watch the answer's speed as much as its content: a vendor who must go ask someone upstream has just told you the collection is not theirs.

Resellers, firms that buy data flows from others and rebadge them, stumble the same way on questions 1, 4, and 6, which only the party operating the collection can answer. Buying from a reseller is sometimes fine, aggregation can be real value, but you should know which kind you are talking to, and the price should reflect it.

Contract terms a measurement can trigger

An SLA, a service level agreement, is the part of the contract that attaches consequences to promises. Most data SLAs promise "high quality" and "timely delivery," which no measurement can trigger, so nothing ever triggers. Rewriting one mostly means importing what this book already built.

• A delivery window with a date, and a completeness manifest with every delivery, the packing slip of a data shipment: file names, row counts per file, checksums, crawl window, delivery date.
• Acceptance defined as the audit: the five checks and their limits, written in as what "delivered" means. You have already rehearsed enforcing them.
• Automatic recredits: rows that fail acceptance, or deliveries that miss the window, convert to credit at the contracted per-row price, no negotiation per incident.
• The quality report arriving with every delivery, not on request.
• Exit portability: at termination you keep the delivered data and manifests under the license, and the vendor owes one final complete delivery. Switching should cost a procurement cycle. Your history stays yours.

A vendor who signs measurable terms is telling you their numbers already clear the bar. Hesitation in front of this list is free information.

One warning before you write check 5 into the acceptance clause: keep the spot list out of it. Write in the mechanism, a buyer-supplied list of 60 items checked against each delivery, but never the items, and draw a fresh list for each acceptance round. The reason sits in the experiment: 213 of Vendor B's 300 fabricated rows pass every row-level check, so the spot list is the only line that catches invention, and a vendor who knows the 60 products in advance can crawl exactly those for real and fabricate the rest, and the audit stops costing cheaters anything. A fresh list is the same afternoon of gathering it took the first time; rotate half of it if a full redraw feels heavy.

Usage rights are a separate purchase

Chapter 2 drew the collection lines and parked one question for this chapter: whether a record was collected lawfully says almost nothing about what you may do with it. Use rights come from your contract, and four clauses do most of the work.

• Internal use versus redistribution. Analysis inside your walls is the default grant. Publishing extracts, reselling, or feeding a customer-facing product each needs its own words.
• Model training. If the data might ever train a model, get the right named, and in writing whether the trained model survives the contract.
• Retention. How long you may keep the data after termination, and what "delete" covers.
• Derived data. Whether aggregates and models built on the data outlive the contract. If your dashboards die with the contract, price that in now.

Chapter 2 already did half of this work. Its eleven-line checklist reverses into a vendor questionnaire: send it, ask for written confirmation line by line, and watch the account and personal-data lines for hesitation. Its hiQ ending doubles as your vetting script, a vendor's legal exposure being your supply risk: ask whether accounts touch your collection path and what happens to your deliveries if a cease-and-desist arrives mid-contract. If training is anywhere in your future, ask for the provenance record that chapter taught collectors to keep, the dated robots.txt stored beside each batch. The vendor who can produce it planned to be asked.

Walking away

Some purchases fail the audit. Others should never reach it. Three exits, each with a document or a number behind it.

The source is hostile. If delivering your specification at the claimed freshness would require what Chapter 2 ruled out, accounts on the collection path or evading a site's defenses, then every vendor quoting you is either quietly under-delivering or carrying a risk that will eventually become your outage. Against a hostile source, I treat the most generous quote as the most alarming document in the folder.

The rights are murky. A vendor who cannot put your intended use in writing is selling you a dispute with a delay on it, and no sample quality changes that.

The economics are negative at real volumes. Rerun Chapter 1's build-versus-buy model with real quotes instead of list prices. That model put the crossing at 316,266 records a month at list, then watched its own answer swing between 106,469 and 746,168 as the vendor's price moved between double list and half, the strongest lever in its sensitivity run. Three written quotes against one specification turn folklore into arithmetic inside a week. And when even the best quote costs more than the decision it feeds is worth, take Chapter 1's quiet third option: no pipeline and no contract, an analyst with a spreadsheet and an afternoon a week, or an official API's free tier. Declining to buy and declining to build is a finding too. File it like one.

That is the book. It opened with a deal: every real question settled by an experiment you can rerun, the prose following the experiment's output rather than my instinct, and wherever a run contradicted me, the run wins. It collected on that promise early, when Chapter 1's model overruled my instinct on the book's first question, and the text kept the correction. Ask a question that has an answer, write down what you expect before you look, run it, keep the numbers beside the code, and say what happened in words the person paying for the work can read. The same loop built every collector in this book and just audited two vendors, and it does not care which chair you sit in. The tools in these pages will age long before the loop does. Check my work. Everything here reruns.