The Data Collection Handbook · Part II. Getting the Data

Chapter 6. Building Resilient Collectors

David Martin Riveros · Published August 3, 2026

The worst data bug I have ever shipped did not crash anything. A collector I ran was pulling prices from a retailer, and one afternoon the retailer started challenging our requests. The collector did what it had been told to do with a failed request: it moved on and wrote down what it got. What it got was a challenge page with no price on it, and the code, finding no price, recorded zero. For most of a day we delivered a feed claiming a whole catalog of products had dropped to nothing. Nobody's program crashed. Every job reported success. The dashboard was green the entire time it was wrong.

That day taught me the discipline this chapter is built around, and it is a way of working more than a clever technique. A collector has to succeed when it can, it has to know when it did not, and it must never, under any pressure, let a failure wear the costume of a fact. Chapters 3 through 5 got you collecting. This one is about the other 15 percent, the share the experiment assumes, the requests that fail, and why designing for that minority rather than the easy majority is what separates a collector you can bill from one that lies to you.

Failure is the normal case

New collectors are written for the happy path, because the happy path is what you test on. You point it at a page, it works, you ship it. Then it meets production, where a real source times out, rate limits you, serves a challenge, returns half a page, or simply has a bad minute. None of that is exotic. It is Tuesday. A collector that treats failure as an exception to be surprised by will be surprised several times an hour, forever.

So flip the design around. Assume every request can fail, and ask a different question: not whether a request failed, but what you are allowed to do with the failure. That question has one dangerous answer. Ruling it out is the job.

The three buckets

Every outcome a request can have sorts into three buckets, and keeping them separate is the habit I care about most in this work.

The first bucket is definitive success. The source answered, the answer is the thing you asked for, and it becomes a data value. Simple.

The second bucket is the one people forget exists, and forgetting it causes its own bugs. It is the definitive negative: the source itself, clearly and on purpose, told you the item is not there. A product page that returns a real not found, not a challenge and not an error, an actual answer that the thing does not exist. That is information, and it is allowed to become a recorded negative, a row that says this item was gone on this date. One rule guards this bucket: only an unmistakable answer from the source earns it.

The third bucket is everything else, and among failures it is by far the largest: the ambiguous failure. A timeout. A block. A challenge page. A rate limit. A response that arrived but did not parse. All of them share one property, and it decides what you may do with them: they tell you nothing about the item. The attempt did not tell you the price is zero or that the product is gone. It only told you that this attempt did not land. The only things an ambiguous failure may become are a retry or an honest gap, a row that says we do not know, recorded as missing rather than invented.

The forbidden edge is the red line in the diagram, and it is the bug from the top of this chapter. An ambiguous failure may never become a data value. A block does not mean a price of zero, a timeout does not mean the product vanished, and a challenge page does not mean out of stock. The moment your code lets any failure fall through into a real value, it has manufactured data, and manufactured data is worse than no data in a way that took me years to feel in my bones: it is billed, it is confident, and nothing downstream can tell it from the truth. A missing row announces itself. A fabricated row hides, gets delivered, gets acted on, and shows up as a decision someone made on a number you invented. Chapter 8 catches these after the fact. Never creating them in the first place is cheaper, and that is the job here.

Retries, and how they go wrong

Once a failure can only become a retry or a gap, the next question is how to retry. The naive answer is: it failed, try again right now, a few times, then give up. It feels responsive. It does real damage, and the experiment below measures how much.

The problem is that the most common reason for a cluster of failures is that the source has started pushing back, and immediate retries push into the push-back. A collector running many requests at once, which is every real collector, responds to a struggling source by instantly sending all its failed requests again, which is a burst on top of the concurrency that was already testing the limit, and the burst is what keeps the block standing. In the experiment below the workers cross the source's limit on their own; what the naive retries add is the habit of spending the whole attempt budget against a door that has already closed. It is a collector fighting itself.

The fix is old and boring and works: exponential backoff with jitter. Backoff means you wait longer after each successive failure, half a second, then one, then two, then four, giving a struggling source room to recover instead of a fresh flood. Jitter means you add a random wobble to each wait, so that a thousand requests that failed together do not all retry at the same instant and re-form the very burst you were trying to avoid.

Timeouts are budgets, not settings

A timeout is how long you wait before deciding a request has failed, and the mistake is having only one, buried deep. If the only timeout is on the individual network request, a collector can still hang far past any deadline: an item that retries ten times, each retry waiting the full per-request timeout, can burn minutes on a single item, and a job made of such items runs as long as items times retries allow, a bound nobody planned for. I have watched a job that was supposed to take an hour still going the next morning, every individual request dutifully under its timeout, the whole making no progress.

The discipline is to budget time at every level. A per-request budget: this single fetch gets N seconds. A per-item budget: all the retries for one item together get M seconds, after which the item is recorded as a gap and the collector moves on. And a per-job budget: the whole run gets a wall-clock ceiling, after which it stops, delivers what it has, and reports the rest as not collected. The per-item budget is the one people miss, and it is the one that turns an infinite hang into a bounded gap. A budget you did not set is a budget set to infinity.

Circuit breakers

Backoff spaces out one item's retries. A circuit breaker handles the case where the whole source has gone bad, and it is the piece that makes a collector stop hurting itself. The idea is a switch with three positions.

Normally the breaker is closed and requests flow. When failures pile up past a threshold, say more than half of recent attempts failing, the breaker opens, and while it is open the collector sends nothing at all to that source for a cool-off period. This is the move that feels wrong and saves the job: in the face of a source that is refusing everything, the most productive thing a collector can do is stop, because every request it sends into a block fails, and on many sources also re-trips the block the moment it lifts. After the cool-off the breaker goes half-open and sends one lone test request. If that succeeds, the source has recovered and the breaker closes back to normal. If it fails, the breaker re-opens and waits again. The breaker is how a collector notices, without a human, that it should back off the whole source rather than keep grinding item by item.

Three policies against one bad source

I wanted to know whether all this politeness costs data, because there is a real intuition that says the aggressive collector, hammering away, must at least collect more of what it can before it gets blocked. So this chapter's experiment pits three policies against the same unreliable source: naive immediate retry, backoff with jitter, and backoff plus a circuit breaker. The source behaves like a real one, a share of requests fail transiently, and sustained hammering trips a block that lasts a while and fails everything underneath it. Twenty workers run at once, the way a real collector does, because it is the concurrency that lets naive retries stampede. The job is 3,000 items, run five times per policy with different random seeds so the numbers are not a fluke. Everything is simulated and seeded, so python run.py reproduces every figure below.

The result was not close, and it did not split the way the aggressive intuition predicts. Naive retry collected 19 percent of the job. Not 19 percent less, 19 percent total: it tripped the block early, then spent the rest of the run hammering a blocked source with immediate retries, most of them failing into the very wall its own bursts kept standing. Backoff with jitter collected 99 percent. Backoff plus a breaker collected 100 percent. The gentle policies did not trade data for manners. They collected five times as much.

Then look at what each policy did to the source. The request counts turn the result from surprising into obvious. Naive retry sent 15,800 requests to collect its miserable 19 percent: 5.27 requests for every item in the job, and about 27 for every item it actually got, most of them wasted against the block. The breaker sent 3,834, about a quarter as many, 1.28 per item with the whole job collected, so for it the two ratios are the same number. The aggressive collector put four times the load on the source and came home with a fifth of the data. It did not trade politeness for yield. It lost on both at once, because against a source that hard-blocks bursts, the load it inflicted is what caused the failures it suffered. A source that only slows you down, rather than shutting the door, narrows that gap without reversing it. So I stopped thinking of resilience and politeness as a trade-off. In this experiment they turned out to be the same property.

One cost to name, because the experiment shows it. The breaker took longer in wall-clock time, because it deliberately waits out its cool-offs instead of hammering: in these runs it spent about seven minutes where naive spent under four. But naive's four fast minutes produced a fifth of the data and a mountain of load, so its speed is the speed of failing quickly. Backoff sat in between, nearly all the data at nearly the lower load without the breaker's patience. Slower is not better in itself. The breaker spends time to buy yield and gentleness, and time is usually the cheapest of the three to spend.

Idempotency, so you can always just re-run

One more property makes all of the above safe to operate: idempotency. A collector is idempotent when running it again with the same input does no damage: no double counts, no double bills, no duplicate rows. The practical version is that each record has a natural key, the identity of the thing, the product code and the date, say, and writing a record means upserting on that key, updating the row if it exists rather than blindly appending a new one. Get this right and recovery becomes trivial: a job died halfway, so you run it again, and the items it already has are simply rewritten under the same keys, with fresher values if the source moved, while the missing ones fill in. Get it wrong and every retry is a risk, and operators start being afraid to re-run the thing. That fear is its own failure: a collector nobody dares restart ends up left broken.

The resilience checklist

I keep this on one page and run every collector against it before it goes near production.

1. Every outcome is classified into definitive success, definitive negative, or ambiguous failure, and the three are handled by separate code paths.
2. No ambiguous failure can produce a data value. A block, timeout, challenge, or parse miss becomes a retry or an honest gap, never a number.
3. A definitive negative is only recorded when the source unmistakably said not found, never inferred from an error.
4. Retries use exponential backoff with jitter, never immediate repetition.
5. Time is budgeted per request, per item, and per job, and the per-item budget exists.
6. A circuit breaker stops all traffic to a source that is refusing, and probes before resuming.
7. On a rate-limit or error signal, the collector slows itself without waiting for a human.
8. The collector is idempotent: safe to re-run on the same input, with records upserted on a natural key.
9. A run reports what it collected and what it could not, and the two never blur.

If I could keep only one line it would be item 2, because it is the one whose absence you cannot see. A collector that breaks most of the others fails where you can see it. A collector that breaks item 2 keeps running, keeps reporting success, and feeds false values into everything downstream; items 3, 8, and 9 guard against the related failures. Everything from here on assumes the data you collected is real. Part III checks that assumption again at the other end, where the data is delivered, and it can only hold if no failure was ever allowed to pose as a fact.