The Data Collection Handbook · Part III. Trusting the Data
Chapter 8. Data Quality and Validation
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
A dataset can arrive on time, in the right format, and still be quietly wrong. This chapter separates the claim we delivered the file from the claim the data is correct, and shows how to earn the second claim with a validator you have actually tested.
What you will learn
- The five measurable dimensions of quality: completeness, validity, consistency, freshness, and plausibility.
- The validation pyramid: schema checks, field rules, cross-field rules, and distribution-level checks, and what each layer catches that the others cannot.
- Field fill rates as the cheapest, most sensitive quality signal a pipeline can watch.
- How to test the tester: plant known errors in clean data and measure how many your validator catches and how many false alarms it raises.
In this chapter
- Delivered is not correct. A shipped dataset with a silent price error nobody downstream can catch.
- The five dimensions. Each defined in one sentence with one concrete failure.
- The validation pyramid. Four layers, from cheap schema checks to distribution checks.
- Fill rates: the master signal. How a breakage shows up as a cliff in one chart.
- Plausibility rules. Encoding domain knowledge as executable checks.
- Testing the tester. Error seeding turns validation from a feeling into a measurement.
- Quality reporting. The one-page report that ships with every delivery.
The experiment
The question: How good is a hand-written validator, really, measured against errors planted at known positions?
A clean, permissively licensed product dataset ships with the chapter. The code plants five realistic error types at known positions, runs a layered validator, and scores it per error type: planted errors caught, false alarms raised. Flags beyond the planted set get inspected rather than dismissed, because the clean data may hold real errors of its own. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that schema-level checks catch almost nothing that matters and the plausibility and distribution layers do the real work. The scorecard will say.