A book in progress

The Data Collection Handbook

By David Martin Riveros · Started July 2026 · 14 chapters, publishing as they are finished

I have spent years building systems that collect web data at scale, and most of what I know arrived the expensive way: through production incidents, wrong datasets, and lessons no book had written down. This handbook is my attempt to write them down, in the open, for the whole industry: the engineers and analysts who build collectors, and the founders and buyers who pay for the results.

Three rules govern every chapter. First, plain language: no jargon without a one-line definition. Second, evidence: each chapter runs one analytical experiment, with data and runnable code published alongside it, so you can replicate every result instead of taking my word. Third, the written analysis carries the full conclusion, so you get the finding even if you never run the code.

One scope note. This book covers collecting publicly visible data at fair load. It does not teach collecting from behind logins, defeating CAPTCHAs, or evading any specific vendor's detection product. Where anti-bot systems appear, the treatment is how detection works and why coherent, respectful traffic is the durable strategy. Those are principled boundaries, not oversights.

This page is the book's index. Each chapter link below leads to that chapter's plan today, and to the full chapter as it publishes. The structure went through independent review from two directions, an engineer's read and a buyer's read, before the first chapter was drafted.

Part I. Foundations

• Chapter 1. The Data Collection LandscapeWho collects web data and why, the five doors into any source, and the honest math of building a collection team versus buying the data.
• Chapter 2. Legality, Ethics, and Fair UsePublic versus gated data, personal data, what robots.txt really is, and the fair-load rule: how to collect without being a bad actor.

Part II. Getting the Data

• Chapter 3. Choosing the Access MethodThe five doors into any source compared on cost, fragility, and completeness, and the waterfall principle for combining them.
• Chapter 4. Network Identity: Proxies and IPsYour IP address is your reputation. Which proxy type fits which job, and how to size a pool with a formula instead of a guess.
• Chapter 5. Looking Human: Fingerprints and Anti-Bot SystemsHow modern sites tell humans from scripts, and why coherent, respectful traffic beats fighting detectors request by request.
• Chapter 6. Building Resilient CollectorsFailure is the normal case. The three-bucket rule, retries done right, and the golden rule: never fabricate a result from a failure.
• Chapter 7. Extraction: From Page to RecordWhere data is actually born. Store the raw response, find the structured data hidden in pages, and build extractors that break loudly, not silently.

Part III. Trusting the Data

• Chapter 8. Data Quality and ValidationDelivered is not the same as correct. Layered validation, field fill rates, and testing your validator by planting known errors.
• Chapter 9. Coverage: Collecting EverythingMissing rows are invisible downstream. Depth walls, segmentation, and how to estimate the true size of a catalog you cannot see.
• Chapter 10. Monitoring, Drift, and HealingSources change without telling you. Detecting silent breakage from the data itself, and alarms that learn what normal looks like.

Part IV. Running It at Scale

• Chapter 11. Cost EngineeringOne number rules collection economics: cost per successfully delivered record. Waterfalls, the four money leaks, and honest unit math.
• Chapter 12. Delivery You Can TrustTruth lives at the destination. Verifying deliveries in storage, re-delivery that is always safe to repeat, and billing that matches what landed.
• Chapter 13. Operating at ScaleFrom one collector to a fleet: queues between stages, concurrency as a promise to the source, rescuing stuck work, and calm incidents.

Part V. The Other Side of the Table

• Chapter 14. How to Buy Data WellThe whole book replayed from the buyer's chair: auditing a sample delivery without code, the seven vendor questions, SLAs, and usage rights.