Noticing the Slide : Calibrated Drift Detection for Deployed Agents

Abstract

A deployed agent rarely breaks all at once. A model update, a changed tool, or a shift in what users ask for can raise its violation rate a little at a time, and nobody notices until the damage is done. We ask how quickly different change detectors notice such drift when each is set to the same false-alarm rate. We compare a CUSUM, a likelihood-ratio CUSUM, an EWMA chart, a rolling-mean threshold, a sliding-window test and an ADWIN-style adaptive window on streams of per-task violation indicators, with every detector calibrated to one false alarm per 1,000 tasks and checked on fresh data. For a step increase from 5% to 10%, the CUSUM family, the EWMA chart and the adaptive window all needed about 69 to 75 tasks (median), the rolling mean 79, and the sliding-window test 120. On bursty drift the gap widens to 4.8 times. For a slow ramp, the classical first-order delay formula was wrong by up to 9 times. We derive a better one, treating the CUSUM as a reflected random walk with slowly changing drift, and it predicts the simulated median delay within 8% over a hundredfold range of slopes. Estimating the baseline from 500 tasks raised false alarms by up to 1.39 times for detectors that use it, and benign seasonality of 100% amplitude raised them about threefold. All detectors ran in under 2 µs per observation. All results are simulation-based.