The Data Collection Handbook · Part IV. Running It at Scale
Chapter 12. Delivery You Can Trust
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
The most expensive lie in this industry is a status field that says delivered. This chapter is about the last mile: verifying at the destination storage instead of trusting your own pipeline's flags, making re-delivery always safe to repeat, and billing only what actually landed.
What you will learn
- The prime rule: truth lives at the destination, in the bucket or warehouse the customer reads, never in your pipeline's status field.
- The status-field failure modes: flags that stay running forever after success, retries that flip a successful delivery to failed, partial writes marked complete.
- Idempotent delivery: deterministic naming and safe repeat semantics, so a re-run can never silently destroy or duplicate.
- Manifests: shipping a small machine-readable summary with every delivery so the customer can verify without trusting you.
- Billing integrity: bill from what landed, not from what was attempted.
In this chapter
- It says delivered. The support ticket that starts with a status field and ends with an empty bucket.
- Verify at the destination. The closed loop that ends arguments.
- Idempotency in the last mile. The retry trap that records a success as a failure.
- Manifests. Files, rows, and checksums the customer can check alone.
- Billing integrity. Reconciling delivered rows against billed units.
- The delivery contract. Destination, naming, format, verification, re-delivery.
The experiment
The question: Under realistic failure injection, how often does a naive status field disagree with what is actually in storage, in both directions?
Ten thousand simulated deliveries to a mock object store with injected faults: network failures, crashes between the upload and the status write, and a blind retry policy. The status table is then compared against ground-truth storage listings, classifying both lie directions: says delivered but is not, and says failed but actually is. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that the naive design lies in both directions at material rates, and that says failed but actually delivered is the nastier direction because it triggers duplicate re-deliveries and double billing. Storage-level verification should drive both directions to zero.