The Data Collection Handbook · Part IV. Running It at Scale
Chapter 11. Cost Engineering
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Cost per request flatters everyone, because failures cost money and deliver nothing. This chapter builds collection economics around the one number that cannot lie, cost per successfully delivered record, and shows how method waterfalls, caching, and honest failure accounting drive it down.
What you will learn
- How to compute true cost per delivered record: infrastructure, proxies, retries, and failures, divided by delivered rows only.
- The waterfall as an economic instrument: cheap methods absorb volume, expensive methods rescue the remainder, and the blended cost emerges from the mix.
- The four classic money leaks: premium infrastructure used on easy pages, re-collecting unchanged data, retrying into permanent failures, and idle always-on capacity.
- Why the optimal mix must be tuned from measured success rates, not from vendor brochures.
In this chapter
- The only number that matters. Why cost per request misleads and cost per delivered record does not.
- Anatomy of the unit cost. One delivered record's cost, line by line.
- The waterfall as an economic instrument. How blended cost emerges from the method mix.
- The four classic leaks. Where collection budgets quietly drain.
- Cost observability. Tagging every request so the blended number is measured, not modeled.
- Pricing data honestly. From unit cost to a price that survives source changes.
The experiment
The question: Given several collection methods with different unit costs and success rates, what ordering and routing minimizes cost per delivered record, and how much does the waterfall save against using the premium method for everything?
A closed-form model of waterfall economics plus a simulation that adds the realistic twist most models skip: difficulty is correlated, so an item that fails on the cheap method is also harder for the next one. The code sweeps orderings and routing rules and quantifies the gap between the naive estimate and the correlated one. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that a tuned waterfall cuts blended cost several-fold against premium-only, and that correlated difficulty erodes the naive savings estimate enough to matter. That erosion is the argument for measuring your own success rates.