The Data Collection Handbook · Part IV. Running It at Scale
Chapter 13. Operating at Scale
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
One collector is a script. Fifty collectors serving paying customers is an operations problem. This chapter covers the fleet disciplines: queues that keep one slow source from stalling everyone, concurrency treated as a deliberate promise to each source, stuck-work recovery, and incident discipline that turns outages into rules.
What you will learn
- Why stages need queues between them, and how backpressure keeps a slow source from taking the system down.
- The two different limits that look alike: what a source tolerates (a promise) and what your fleet can afford (a budget).
- How to make sure nothing sleeps forever: heartbeats, timeouts on invisible work, and automatic rescue of stuck jobs.
- Fair scheduling across customers, and why first-come-first-served starves someone.
- Incident discipline: severity, stop the bleeding first, and blameless write-ups.
In this chapter
- The night everything queued. One slow source, every customer stalled.
- Queues between stages. Accept, schedule, collect, process, deliver as independent stages.
- Concurrency is a promise. Per-source ceilings versus global capacity.
- Nothing sleeps forever. Heartbeats, timeouts, and automatic rescue.
- Scheduling the fleet. Priorities, fairness, off-peak windows.
- Incident discipline. The first three moves, and the write-up that prevents the repeat.
- The operations maturity checklist. From a person watches it to the system watches itself.
The experiment
The question: With realistic bursty job arrivals, what is the smallest worker fleet that keeps 95 percent of jobs starting within the promised wait time, and how wrong is the answer if you assume smooth arrivals?
A queueing model as the baseline intuition, then a discrete-event simulation with bursty arrivals. The code sweeps worker counts, measures typical and worst-case waits and fleet cost, and compares the bursty answer against the smooth-arrivals prediction. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that burstiness, not average load, sets the fleet size, and that the smooth-arrivals model undersizes the fleet badly at exactly the promise customers feel: the worst-case wait.