The Data Collection Handbook · Part III. Trusting the Data
Chapter 10. Monitoring, Drift, and Healing
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
The site you collect from will redesign on a Tuesday and your collector will keep returning success while shipping garbage. This chapter is about catching that silently, from the data itself, with alarms that learn each signal's normal wobble from its own history.
What you will learn
- The three failure surfaces: infrastructure (it crashed), access (you are blocked), and semantic drift (everything runs green but the data is wrong), and why each needs its own detector.
- The signals that catch semantic drift cheaply: per-field fill rates, row counts against baseline, value distributions, duplicate rates.
- Alert design as a real trade-off: fixed thresholds versus rules that learn from history, including the family built to catch slow drifts by accumulating small deviations.
- The baseline discipline: never alarm without history, and how much history an alert needs before you can trust it.
- The healing loop: from alert to verified fix, with sentinels so the same break cannot return silently.
In this chapter
- The site redesigned on a Tuesday. Two weeks of green dashboards over broken data.
- Three failure surfaces. Infrastructure, access, semantic: each invisible to the others' detectors.
- Signals from the data itself. Fill rates, counts, distributions, freshness lag.
- Alert design. Fixed thresholds, history-aware rules, and the slow-drift specialists.
- Baseline discipline. Never alarm without history.
- The healing loop. Detect, diagnose, fix, verify with the same monitor that caught it.
The experiment
The question: When a breakage degrades a field gradually, which alerting rule catches it fastest without crying wolf: a fixed threshold, a classic control chart, or an accumulating rule built for slow drifts?
Ninety days of a field's fill rate, simulated with realistic noise and weekly seasonality, with a gradual break injected at a known day. The three rules run over hundreds of randomized trials, measuring mean detection delay and false alarms per month for each. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that the accumulating rule wins on gradual drifts by collecting many small deviations into one confident alarm. Whatever ranking the trials produce, the durable lesson stands: monitoring quality is a statistics problem, not a dashboards problem.