The Data Collection Handbook · Part I. Foundations
Chapter 1. The Data Collection Landscape
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Every data project starts with the same three questions: what is out there, how do people get it, and should we collect it ourselves or buy it? This chapter maps the industry so you can answer all three without guessing.
What you will learn
- Who collects web data and for what: pricing intelligence, market research, AI training, monitoring.
- The five doors into any source, one canonical list used through the whole book: the official API, the page HTML, the site's own hidden JSON API, the mobile app's API, and a full browser.
- The option that skips collection entirely: buying ready-made data, and how to treat that as a build-versus-buy decision rather than a reflex.
- What an in-house collection capability really costs, including the part most teams forget: the maintenance tax and the people.
In this chapter
- Data is the raw material of decisions. Two companies pricing the same product, one flying blind.
- Who collects, and for what. The main industry verticals with one concrete scenario each.
- The five doors into any source. A first look at the access methods; Chapter 3 goes deep.
- Build vs buy. The honest cost of an in-house team against vendor pricing, and when the answer is neither.
- The maturity ladder. From a script on a laptop to a monitored data product.
The experiment
The question: At what monthly volume of collected records does building in-house become cheaper than buying from a vendor?
A transparent cost model: engineer time, proxy and compute cost, and a maintenance tax that grows with every source added, against parameterized vendor price tiers. The model sweeps volume from ten thousand to one hundred million records per month and finds the breakeven point, with a sensitivity analysis on the most uncertain inputs. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My working hypothesis is that the breakeven sits much higher than most teams assume, and that the maintenance tax, not the initial build, is what drives it. The published chapter will report whatever the model actually shows.