The Data Collection Handbook · Part II. Getting the Data
Chapter 3. Choosing the Access Method
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Most teams default to scraping the rendered page when the site itself often ships a cleaner, faster data source to its own frontend. This chapter teaches the fifteen-minute audit that finds every door into a source, and how to order those doors into a waterfall so each record is collected the cheapest reliable way.
What you will learn
- How to audit a website in fifteen minutes and find all five doors: official API, page HTML, the hidden JSON API feeding the frontend, the mobile app's API, and the full browser.
- How to score each door on the three axes that matter: cost per record, fragility, and completeness.
- The waterfall principle: try the cheap method first, fall through to expensive ones only on failure, and log which door served every record.
- When a full browser is genuinely the right answer, and why it is always the last resort.
In this chapter
- The default is wrong. Why rendered-page scraping is rarely the best first choice.
- The 15-minute source audit. A practical walkthrough of finding every door a site exposes.
- The five doors compared. One honest table: cost, fragility, completeness, effort.
- The waterfall principle. Cheap methods absorb volume, expensive methods rescue the remainder.
- When the browser is the right answer. The minority of cases that need full rendering.
The experiment
The question: On the same public source, how much faster, lighter, and more stable is the site's JSON data door compared to scraping its HTML?
The same 500 items collected two ways from a public source that exposes both doors, using a list-style JSON API against the rendered HTML pages, with both responses shipped as a dated snapshot. The comparison measures request latency, bytes transferred, request count, extraction code size, and how many page selectors the HTML path depends on. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that the JSON door is several times faster and roughly an order of magnitude lighter, and that it cannot break from a layout redesign. APIs break in their own ways, through deprecations and moved endpoints, and the chapter will treat that trade honestly.