The Data Collection Handbook · Part II. Getting the Data

Chapter 6. Building Resilient Collectors

David Martin Riveros · Published August 3, 2026

The worst data bug I have ever shipped did not crash anything. A collector I ran was pulling prices from a retailer, and one afternoon the retailer started challenging our requests. The collector did what it had been told to do with a failed request: it moved on and wrote down what it got. What it got was a challenge page with no price on it, and the code, finding no price, recorded zero. For most of a day we delivered a feed claiming a whole catalog of products had dropped to nothing. Nobody's program crashed. Every job reported success. The dashboard was green the entire time it was wrong.

That day taught me the thing this chapter is built around, and it is not a clever technique. It is a discipline. A collector's job is not only to succeed. It is to know the difference between succeeding and failing, and to never, under any pressure, let a failure wear the costume of a fact. Chapters 3 through 5 got you in the door. This chapter is about what happens on the 15 percent of requests where the door sticks, and why designing for that 15 percent, not the easy 85, is what separates a collector you can bill from one that quietly lies.

Failure is the normal case

New collectors are written for the happy path, because the happy path is what you test on. You point it at a page, it works, you ship it. Then it meets production, where a real source times out, rate limits you, serves a challenge, returns half a page, or simply has a bad minute. None of that is exotic. It is Tuesday. A collector that treats failure as an exception to be surprised by will be surprised several times an hour, forever.

So flip the design around. Assume every request can fail, and make the interesting question not whether a request failed but what you are allowed to do about it. That question has exactly one dangerous answer, and ruling it out is the whole game.

The three buckets

Every outcome a request can have sorts into three buckets, and keeping them separate is the single most important habit in collection.

The first bucket is definitive success. The source answered, the answer is the thing you asked for, and it becomes a data value. Simple.

The second bucket is the one people forget exists, and forgetting it causes its own bugs. It is the definitive negative: the source itself, clearly and on purpose, told you the item is not there. A product page that returns a real, genuine not found, not a challenge, not an error, an actual answer that the thing does not exist. That is information, and it is allowed to become a recorded negative, a row that honestly says this item was gone on this date. Chapter 2's neighbor to this rule matters here: only an unmistakable answer from the source earns this bucket.

The third bucket is everything else, and it is the largest: the ambiguous failure. A timeout. A block. A challenge page. A rate limit. A response that arrived but did not parse. Every one of these has the same defining property, and it is the property that decides what you may do with it: it tells you nothing about the item. You did not learn that the price is zero or that the product is gone. You learned that this attempt did not land. The only things an ambiguous failure may become are a retry or an honest gap, a row that says we do not know, recorded as missing rather than invented.

Now the forbidden edge, the red line in the diagram, the bug from the top of this chapter. An ambiguous failure may never become a data value. A block is not a price of zero. A timeout is not a product that vanished. A challenge page is not an out of stock. The moment your code lets any failure fall through into a real value, it has manufactured data, and manufactured data is worse than no data in a way that took me years to feel in my bones: it is billed, it is confident, and nothing downstream can tell it from the truth. A missing row announces itself. A fabricated row hides, gets delivered, gets acted on, and shows up as a decision someone made on a number you invented. Chapter 8 is about catching these after the fact. This chapter is about never creating them in the first place, which is cheaper.

Retries, and how they go wrong

Once you have accepted that a failure can only become a retry or a gap, the question is how to retry, and the naive answer makes everything worse. The naive answer is: it failed, try again right now, a few times, then give up. It feels responsive. It is actively harmful, and this chapter's experiment measures exactly how harmful.

The problem is that the most common reason for a cluster of failures is that the source has started pushing back, and immediate retries push into the push-back. A collector running many requests at once, which is every real collector, responds to a struggling source by instantly sending all its failed requests again, which is a burst, which is precisely the hammering that trips a block. The retries cause the condition the retries are reacting to. It is a collector fighting itself.

The fix is old and boring and works: exponential backoff with jitter. Backoff means you wait longer after each successive failure, half a second, then one, then two, then four, giving a struggling source room to recover instead of a fresh flood. Jitter means you add a random wobble to each wait, so that a thousand requests that failed together do not all retry at the same instant and re-form the very burst you were trying to avoid. Wait longer, and wait raggedly.

Timeouts are budgets, not settings

A timeout is how long you wait before deciding a request has failed, and the mistake is having only one, buried deep. If the only timeout is on the individual network request, a collector can still hang effectively forever: an item that retries ten times, each retry waiting the full per-request timeout, can burn minutes on a single item, and a job made of such items has no bound on how long it runs. I have watched a job that was supposed to take an hour still going the next morning, every individual request dutifully under its timeout, the whole making no progress.

The discipline is to budget time at every level, like nested envelopes of money. A per-request budget: this single fetch gets N seconds. A per-item budget: all the retries for one item together get M seconds, after which the item is an honest gap and the collector moves on. And a per-job budget: the whole run gets a wall-clock ceiling, after which it stops, delivers what it honestly has, and reports the rest as not collected. The per-item budget is the one people miss, and it is the one that turns an infinite hang into a bounded gap. A budget you did not set is a budget set to infinity.

Circuit breakers

Backoff spaces out one item's retries. A circuit breaker handles the case where the whole source has gone bad, and it is the piece that makes a collector stop hurting itself. The idea is a switch with three positions.

Normally the breaker is closed and requests flow. When failures pile up past a threshold, say more than half of recent attempts failing, the breaker opens, and while it is open the collector sends nothing at all to that source for a cool-off period. This is the counterintuitive move that saves the job: in the face of a source that is refusing everything, the most productive thing a collector can do is stop, because every request it sends into a block both fails and prolongs the block. After the cool-off the breaker goes half-open and sends one lone test request. If that succeeds, the source has recovered and the breaker closes back to normal. If it fails, the breaker re-opens and waits again. The breaker is how a collector notices, without a human, that it should back off the whole source rather than keep grinding item by item.

The experiment: gentler wins

I wanted to know whether all this politeness costs data, because there is a real intuition that says the aggressive collector, hammering away, must at least collect more of what it can before it gets blocked. So this chapter's experiment pits three policies against the same unreliable source: naive immediate retry, backoff with jitter, and backoff plus a circuit breaker. The source behaves like a real one, a share of requests fail transiently, and sustained hammering trips a block that lasts a while and fails everything underneath it. Twenty workers run at once, the way a real collector does, because it is the concurrency that lets naive retries stampede. The job is 3,000 items, run five times per policy with different random seeds so the numbers are not a fluke. Everything is simulated and seeded, so python run.py reproduces every figure below.

The result was not close, and it did not split the way the aggressive intuition predicts. Naive retry collected 19 percent of the job. Not 19 percent less, 19 percent total: it tripped the block early, then spent the rest of the run hammering a blocked source with immediate retries, most of them failing into the very wall its own bursts kept standing. Backoff with jitter collected 99 percent. Backoff plus a breaker collected 100 percent. The gentle policies did not sacrifice data for manners. They collected five times as much of it.

Then look at what each policy did to the source, because this is where the lesson turns from surprising into obvious. Naive retry sent 15,800 requests to collect its miserable 19 percent: 5.27 requests for every item it actually got, most of them wasted against the block. The breaker sent 3,834, about a quarter as many, 1.28 per collected item, and got the whole job. The aggressive collector inflicted four times the load and came home with a fifth of the data. It was not trading politeness for yield. It was worse on both, at once, because the two are the same thing: the load it inflicted is what caused the failures it suffered. Resilience and politeness are not a trade-off you balance. They are one property with two names.

One honest cost to name, because the experiment shows it and hiding it would be exactly the dishonesty this chapter argues against. The breaker took longer in wall-clock time, because it deliberately waits out its cool-offs instead of hammering: in these runs it spent about seven minutes where naive spent under four. But naive's four fast minutes produced a fifth of the data and a mountain of load, so its speed is the speed of failing quickly. Backoff sat in between, nearly all the data at nearly the lower load without the breaker's patience. The right reading is not that slower is better. It is that the breaker spends time to buy both yield and gentleness, and time is usually the cheapest of the three to spend.

Idempotency, so you can always just re-run

One more property makes all of the above safe to operate, and it is the quiet one: idempotency. A collector is idempotent when running it again with the same input produces the same result without doing damage, so that a re-run never double counts, double bills, or double writes. The practical version is that each record has a natural key, the identity of the thing, the product code and the date, say, and writing a record means upserting on that key, updating the row if it exists rather than blindly appending a new one. Get this right and recovery becomes trivial: a job died halfway, so you run it again, and the items it already has are simply rewritten with the same values while the missing ones fill in. Get it wrong and every retry is a risk, every recovery a chance to corrupt, and operators start being afraid to re-run the thing, which is its own kind of failure. A collector you are afraid to restart is a collector that will eventually be left broken because restarting it felt too dangerous.

The resilience checklist

I keep this on one page and run every collector against it before it goes near production.

1. Every outcome is classified into definitive success, definitive negative, or ambiguous failure, and the three are handled by separate code paths.
2. No ambiguous failure can produce a data value. A block, timeout, challenge, or parse miss becomes a retry or an honest gap, never a number.
3. A definitive negative is only recorded when the source unmistakably said not found, never inferred from an error.
4. Retries use exponential backoff with jitter, never immediate repetition.
5. Time is budgeted per request, per item, and per job, and the per-item budget exists.
6. A circuit breaker stops all traffic to a source that is refusing, and probes before resuming.
7. On a rate-limit or error signal, the collector slows itself without waiting for a human.
8. The collector is idempotent: safe to re-run on the same input, with records upserted on a natural key.
9. A run reports what it collected and what it honestly could not, and the two never blur.

Item 2 is the one I would keep if I could keep only one, because it is the one whose absence you cannot see. A collector that violates any of the others fails loudly. A collector that violates item 2 keeps running, keeps reporting success, and poisons the well. Which is the bridge to the rest of this book: everything from here on assumes the data you collected is real, and that assumption is only as good as the discipline that a failure was never allowed to pose as a fact. Making sure of that, at the point of delivery rather than the point of collection, is what Part III is about.