The Data Collection Handbook · Part II. Getting the Data
Chapter 6. Building Resilient Collectors
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
A production collector lives in a world of timeouts, blocks, and half-loaded pages. This chapter is about designing for that world, and about the single worst data bug in the industry: converting a failure into fake data, like recording a blocked request as product not found.
What you will learn
- The three buckets every outcome falls into: definitive success, definitive negative (the source itself said not found), and ambiguous failure (timeout, block, parse miss).
- The golden rule: only a definitive answer from the source may produce a data value; an ambiguous failure may only produce a retry or an honest gap.
- The three protection layers, each in one line: retries with backoff and jitter (wait longer after each failure, with a random wobble), timeout budgets (a time allowance per request, item, and job), and circuit breakers (a switch that stops all attempts when failures pile up).
- Idempotency: building collectors whose runs are always safe to repeat.
In this chapter
- Failure is the normal case. A production collector's honest outcome distribution.
- The three buckets. The classification that prevents fabricated data.
- Retries done right. Why naive immediate retries make things worse.
- Timeouts as budgets. Per-request, per-item, per-job.
- Circuit breakers. Stop asking a source that is telling you no.
- Idempotency. Same input, same output, safe to re-run.
- The resilience checklist. One page to review any collector against.
The experiment
The question: Which retry policy delivers the most successes for the least load on the source: naive retry, backoff with jitter, or backoff plus a circuit breaker?
A simulation of a source with transient failures, a rate limit, and a block state triggered by hammering. Three client policies run over thousands of requests; the code measures final success rate, total requests sent (the load you inflict), and time to completion. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that naive retries cause the very blocks they then suffer from, and that backoff plus a breaker wins on success rate while sending materially fewer requests. Resilience and politeness are the same engineering, and the numbers should show it.