The Data Collection Handbook · Part III. Trusting the Data
Chapter 9. Coverage: Collecting Everything
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
The most dangerous defect in collected data is the row that is not there, because nothing downstream can see an absence. This chapter is about the four ways rows go missing, the segmentation techniques that recover them, and a statistical trick borrowed from ecology that estimates how much you did not get.
What you will learn
- Why missing rows are worse than wrong rows, and why no validator can catch them.
- The four coverage killers: pagination depth walls, hard result caps, orderings that shift between pages, and over-aggressive deduplication.
- Segmentation: slicing a source into facets small enough that each slice fits under its caps.
- Capture-recapture estimation: how two independent partial crawls reveal the true size of the whole.
- How to state coverage honestly to a data buyer: scope, method, estimated completeness.
In this chapter
- The silent killer. A feed that looks perfect and is missing a third of the catalog.
- Where rows go missing. The four failure modes drawn on one paginated catalog.
- Segmentation. Breaking a capped catalog into slices that fit.
- Counting what you cannot see. The ecologists' two-sample trick, translated.
- Coverage verification in production. Reconciling against the source's own counts.
- The coverage contract. Stating completeness honestly in a data deal.
The experiment
The question: Using only two partial, independent crawls of a catalog, how accurately can you estimate its true size without ever seeing the full list?
A simulated catalog of known size, so the truth is available for checking. Two independent partial crawls with realistic bias, a standard two-sample estimator with a confidence interval, and a sweep over sample sizes and bias levels to see where the estimate holds and where it degrades. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that two modest independent samples bound a catalog's size surprisingly well, and that the estimator honestly degrades when the two crawls stop being independent, which is exactly why they must differ in ordering and segmentation.