The Data Collection Handbook · Part IV. Running It at Scale

Chapter 12. Delivery You Can Trust

David Martin Riveros · Published August 21, 2026

The customer writes that Monday's files are not in the bucket. Your own status table says delivered. Now there are two accounts of one bucket, and one of them is wrong, and it is probably yours, because the customer is reading the actual storage while you are reading a note your own code wrote to itself. That three-sentence ticket is the most expensive one in this business. I have answered a few. They start as missing files and become an argument about whether anything your system says can be believed.

Chapter 11 made cost per delivered record the number that runs a collection business. Every part of that calculation leans on one word staying true. Delivered, according to what? In most pipelines the real answer is a status field the pipeline asserted about itself, and this chapter measures how often that assertion is false. The last mile has its own disciplines. Verify at the destination instead of trusting your own flags. Make re-delivery safe to repeat. Ship proof the customer can check without you. Bill only what landed. Four small habits, and the rest of the chapter puts numbers on what ignoring them costs.

Where the drift is born

A delivery pipeline keeps a small record per delivery, the status row, saying running, delivered, or failed. The customer's storage holds the actual files, usually in an object store, a service such as S3 or Google Cloud Storage that keeps files in named containers called buckets. The row and the bucket are written by different actors, at different moments, across a network that fails. That gap is where they drift apart. Given enough deliveries, the two will disagree.

Three mechanics do most of the damage. A process can die in the boundary between the last upload and the status write, and the delivery it just completed stays at running forever. An upload can time out ambiguously: the client gave up waiting, but the bytes may have landed anyway, and nothing visible to the client separates the two cases. And a loop can reach its final line with a partial result and mark delivered anyway, because the code finished even though the data did not.

You do not need a bad engineer for any of this. A network, a deadline, and a pipeline that treats its own memory of events as reality will produce the same drift.

Truth lives at the destination

The rule of the last mile is short enough to put on a wall: truth lives at the destination, in the bucket the customer reads, and in neither your database nor the green tile on your dashboard. A status field is a claim. The bucket is a fact.

Living by the rule means closing a loop most pipelines leave open. After the last upload, list the destination: ask the store which objects exist under this delivery's prefix, the folder-like path its files share, and at what sizes. Compare the listing against what you meant to send, file by file, and write the status only then, as a reading of that comparison. The row becomes a cache of something observed, and anything cached from storage can be re-derived from storage the moment a doubt appears. Everything else in the last mile leans on that. When the three sentence ticket arrives, you do not scroll logs reconstructing the night from memory; you run the same listing the customer can run, and the conversation ends in minutes, whichever way it goes.

The retry that records a success as a failure

Verification tells you a delivery is incomplete. Whether you can fix it safely is a separate discipline, and its name is idempotency: an operation is idempotent when running it twice leaves the world exactly as running it once did. In the last mile that comes down to naming. Give every file a deterministic name, computed from the delivery and the part, orders_2026-08-19_part-2.csv, never anything containing a random job id. A re-run then writes the same names it wrote before, replaces its own partial leftovers, and cannot mint a second copy of anything.

Plenty of pipelines do the opposite twice over. They name files after the run, so each retry creates new objects beside the old ones. And some opt in to create-only mode, where the store refuses to overwrite an existing object. Create-only sounds prudent, and it hides a trap with teeth.

Walk the sequence. An upload times out. The pipeline cannot know the bytes landed, so it does the reasonable thing and retries. The store answers already exists, an error the retry logic never anticipated, and code meeting an unanticipated error does what unhandled paths always do: it aborts and marks the whole delivery failed. Every row is in the customer's bucket, and your table says failed. Ops, doing its job, will now re-deliver everything under a fresh job id, landing every row twice. One ambiguous timeout has become duplicate data and a wrong invoice, and the bug is nowhere you would think to look.

Ten thousand deliveries, four kinds of day

I wanted to know how often this actually happens, so I built the sloppy pipeline on purpose and measured it. The chapter's example simulates deliveries to a mock object store, 10,000 per scenario, each delivery 3 files of 1,000 rows, so one scenario moves 30,000,000 rows. Every upload attempt can fail cleanly, time out ambiguously with the write landing anyway at probability 0.5, or die in a crash at any step boundary, including the one between the last upload and the status write. Four fault environments span a factor of twenty on clean upload failures, from a calm day at 1% to an incident day at 20%. Those rates are my modeling choices, not measurements of any provider. Each one lives in the example's data/params.json with a one line justification, and you can change them and re-run; everything is offline and seeded, so python run.py reproduces every number in this chapter.

Two pipelines face identical deliveries under identical pre-drawn fault schedules, with the same budget of 2 upload attempts per file. The naive one behaves exactly as the last two sections warned: create-only writes, a blind retry, anticipated errors skipped, delivered asserted at the end of the loop, re-delivery under a fresh job id. The verified one writes idempotent names, settles its status from a bucket listing, and reconciles crash victims from storage. Afterward an ops pass re-runs everything marked failed, because that is what a team that trusts its status field does.

Scoreboard after the first pass, in the typical month scenario: the naive status row disagreed with the bucket on 142 of 10,000 deliveries (1.42%). The breakdown: 43 rows said delivered while data was missing. 71 said failed while every row was present. 117 sat at running forever, 28 of them over complete data, and only those 28 join the lie count, because a running row over an unfinished bucket is at least telling the truth. All of that on an unremarkable month.

The fault dial moves the rate and never removes it. On the calm day the flag still misled on 45 of 10,000 deliveries (0.45%), one misleading delivery per 222 on a healthy network. On the incident day it misled on 2,021 (20.21%). And the verified pipeline, under the same pre-drawn faults: 0 of 10,000. In every scenario, in both directions, both before and after re-delivery. It did not lie at all, because a status computed from the bucket has nothing to drift from.

The bill for trusting the flag

Disagreement counts feel abstract until the ops pass runs. In the typical month the team re-ran the 228 deliveries the naive flag called failed, and 71 of those re-runs were unnecessary, deliveries whose every row was already sitting in the bucket. Because re-delivery wrote under fresh job ids, the blind re-runs landed 444,000 duplicate rows in the customer's bucket, 1.48% of the whole feed. Under per-row billing that is double billing, and under per-delivery billing it is corruption, rows the customer's loader ingests twice.

The invoice came out wrong in both directions at once: billed from the status table, it charged for 44,000 rows that never landed and skipped 189,000 rows that landed without ever being marked delivered. Which direction hurts more is not close. The 189,000 unbilled rows are revenue you gave away, and you will never even know the size of the gift unless you reconcile. The 44,000 phantom rows are the ones a customer's auditor finds, and after that audit every invoice you send gets re-checked line by line.

Re-delivery healed selectively, which is the finding that surprised me most in the whole run. The pass cut says failed with data complete from 71 down to 3, at the price of the duplicates, and even that residue has churn inside it: tracked delivery by delivery, 67 healed, two re-runs crashed into stuck-running, two stayed put, and the pass minted one brand-new hidden success by walking into the same create-only trap. It cured 0 of the 43 phantom deliveries. Says delivered with data incomplete stayed 43 of 43, because no process anywhere re-checks a delivery marked delivered. The phantom is permanent until the customer finds it. The stuck-running rows did not heal either; nobody re-runs those, and only a reconciler that reads the bucket, the verified pipeline's habit, ever settles them.

The permanence has a dashboard consequence. A tile that counts status rows can never show a phantom, because a phantom is marked delivered, and alerts keyed to failed stay quiet for the same reason. Make the green tile count bucket listings that matched rather than rows that say delivered, and treat any delivered number that nothing re-checks as testimony. A monthly storage query, the same one the billing section will ask for, settles the delivered tile too; put it on a schedule, and phantoms stop being permanent.

One result decided the design for me: it delivered more data, on top of the cleaner bookkeeping. End state in the typical month, 9,995 complete deliveries of 10,000 against the naive pipeline's 9,863. On the incident day, 9,612 against 7,464, under identical faults. No extra retries were granted anywhere. The verified pipeline spent its ops pass on the right deliveries, because a failed list read from the bucket is a correct repair list, while the naive team re-ran 71 healthy deliveries as its 43 phantoms sat marked delivered.

Those end states are what I read before signing a service level agreement, the completeness promise written into a data contract. Under identical faults and the same retry budget, the naive pipeline truly completed 98.63% of deliveries in the typical month, so a 99% promise breaks on an unremarkable month, and its delivered tile still includes 43 deliveries with data missing, so the customer reports the breach before you do. The verified pipeline finished the same month at 99.95% and held 96.1% on the incident day, against 74.6%. My rule: promise only a completeness number a bucket listing can prove, and never sign 99% with a last mile that asserts its own status.

Judging my own hypothesis

The chapter plan I published stated the hypothesis before anything ran: the naive design lies in both directions at material rates; the nastier direction is says failed but actually delivered, because it triggers duplicate re-deliveries and double billing; and storage-level verification drives both directions to zero.

Two of those three claims survived contact with the data cleanly. The rates are material everywhere, 0.45% of deliveries misled on the calm day rising to 20.21% on the incident day, both directions present in every scenario. The zero is exact, 0 of 10,000 in every scenario, both directions, both measurement points. The middle claim needed a correction I did not see coming. At calm and typical fault rates the hidden success direction is the bigger problem, 71 deliveries against 43, and the 211,000 duplicate rows its unnecessary re-runs landed against 44,000 overbilled. The pass's other 233,000 duplicates came from re-running genuinely incomplete deliveries, a cost of the fresh-name design rather than of either lie. Push the fault dial to incident levels and the balance flips harder than the raw counts suggest: 1,572 phantoms against 254 hidden successes, and 1,740,000 rows overbilled against the 667,000 duplicates the hidden successes caused. Which lie is nastier turns out to track how bad your day is. Permanence does not flip. Re-delivery cut the hidden successes from 71 to 3 and repaired 0 of 43 phantoms, so on any kind of day, the lie that says delivered is the one that never heals. I predicted one villain and the data handed me two, taking turns.

A manifest the customer can check alone

Everything so far still runs on your side of the fence, and a customer who has been burned once is entitled to ask why your verification deserves more faith than your old status field did. The durable answer is to hand over the means of verification. Alongside every delivery, ship a manifest: a small machine-readable file stating what the delivery should contain, the file names, a row count per file, and a checksum per file, a checksum being a short fingerprint computed from a file's bytes that changes if a single byte changes. Write the manifest last, after every data file, so its presence marks a finished delivery and a loader that waits for it can never ingest a half-written one.

The customer's side then shrinks to a small script: list the bucket, count rows, compare checksums. A match accepts the delivery with nobody trusting anybody. A mismatch names the exact files to re-send, which turns re-delivery from an argument into a work order. Months later this is also what keeps disputes short. Both sides compute the same answer from the same bucket, and what remains to argue about is arithmetic.

Billing from the bucket

An invoice is a status field with money attached, so treat it with the same suspicion and bill from what landed, never from what was attempted. Concretely, the billing job reads the verified statuses, or better, the bucket listings themselves, and never touches the raw status table. In the run, the verified pipeline's ops pass made 134 re-runs, every one needed, landed 0 duplicate rows, and produced an invoice that matched storage to the row: 29,985,000 rows billed, and the same 29,985,000 verified present for the billed deliveries. The 5 deliveries that could not complete were billed at nothing and reported as failures, their 10,000 rows of partial data named rather than hidden. Reporting your own failures with that precision costs some pride. A customer who believes those failure reports will also believe your invoices.

Reconcile monthly anyway, delivered rows against billed units, computed from storage. It is one query, and this chapter's run just priced what it catches.

The delivery contract

None of this discipline survives being informal, so agree it in writing with each customer before the first delivery. One page. Mine carries six lines.

1. Destination. The exact bucket and prefix, who owns them, and which credentials may write there.
2. Naming. The deterministic pattern for every file, date and part number in the name, nothing random.
3. Format. File format, compression, encoding, and how corrections are represented.
4. Manifest. Its name, its fields, and the promise that it is written last.
5. Verification. The check each side runs, and the agreement that the bucket plus the manifest settles any disagreement, not either party's dashboard.
6. Re-delivery. Who may request it, how, and the guarantee that it rewrites the same names and can never duplicate.

The page reads like bureaucracy until the first bad night. Then it is why that night stayed boring.

If you are on the buying side, this page is your checklist read in reverse, and there are two questions I would put to any vendor before signing. First, how is delivered computed: from a listing of my bucket, or from your pipeline's own flags? In this chapter's run the flag was wrong on 142 of 10,000 deliveries in the typical month. Second, does a re-delivery rewrite the same file names? The vendor who answers no is the one whose routine re-runs landed 444,000 duplicate rows in the customer's bucket. Make lines 4 through 6 the vendor's written obligations. The full buyer's audit arrives in Chapter 14; these two questions open the door.

It also assumes something larger than any one delivery: that repeating work is always safe. Running a fleet of collectors spends that assumption everywhere, because every rescue of a job that died with work half done is a re-run someone must be able to fire at three in the morning without fear. You just made repetition harmless. That is what lets the fleet stay calm.