Self-Censoring Anomaly Thresholds: Truncation-Corrected Noise Floors for Streaming Detection

Streaming anomaly detectors turn a score into an alarm by comparing it to a threshold that follows the local noise level. Engineers learned long ago that anomalies inside the tracking window inflate that noise estimate and hide later anomalies, a problem called threshold poisoning. The usual fix is to update the baseline only with points the detector did not flag. This proposal argues that the fix creates a second, quieter error. A window that excludes everything above the current threshold is a truncated sample: its tails have been cut off by the detector itself. Its variance therefore underestimates the true noise floor. The threshold then tightens, more points are excluded, and the estimate settles at a biased value. Daskalakis and colleagues proved that this plug-in feedback loop breaks variance estimation in truncated regression (Efficient Truncated Linear Regression with Unknown Noise Variance). As far as the file shows, nobody has examined it inside anomaly thresholding, where the truncation boundary is set by the detector itself.

The core idea fits in two sentences. When a detector updates its noise estimate only from points it accepted, it should treat those points as a sample truncated at a boundary it knows exactly, and estimate the noise scale by truncated maximum likelihood rather than by sample variance. Dividing scores by this corrected, time-varying scale and then fitting an extreme-value tail to the result should give alarm rates that match the operator's target risk even when the physical noise floor drifts. Uncensored rolling statistics do not deliver this, and neither does naive censoring.

Core Claim

The claim has two parts, and the first can be tested on its own. The first part is that censored-window updates produce a measurable, systematic miscalibration. Consider a Gaussian bulk censored at $k$ times its estimated scale. Iterating "estimate the scale from accepted points, reset the cutoff" converges to a fixed point below the true scale, and the fixed point sinks further as $k$ shrinks. At $k=2$ the retained variance is roughly three quarters of the truth before iteration. At $k=3$ the shortfall is only a few percent, but at the extreme risk levels operators use (for example $10^{-4}$) a few percent of scale error inflates the realized false-alarm rate by tens of percent. The file reports no measurement of this effect in anomaly pipelines, so establishing its size on real benchmarks is itself a contribution.

The second part is the remedy. Write the accepted residuals in a window as $r_i$, observed only when they fall in a known set $S_t$ (for example $[-\tau_{t-1}, \tau_{t-1}]$, possibly intersected with the sensor's clipping range). We estimate mean $\mu_t$ and scale $\sigma_t$ by maximizing the truncated likelihood in the canonical parameters $v=\mu/\sigma^2$ and $\lambda=1/\sigma^2$. Daskalakis et al. show that this parameterization makes the population log-likelihood convex, and they solve it by projected stochastic gradient descent started from the biased plug-in estimate. We reuse exactly that estimator, warm-started from the previous window. The standardized score $z_t=(r_t-\hat\mu_t)/\hat\sigma_t$ is then passed to a peaks-over-threshold tail fit in the style of SPOT. The alarm threshold becomes

$$\tau_t = \hat\mu_t + \hat\sigma_t\, z_q,$$

where $z_q$ is the quantile of the fitted Generalized Pareto tail (the limiting distribution of exceedances over a high level) at the user's risk $q$. The design separates two jobs. The time-varying scale absorbs noise-floor changes, and the tail model on standardized scores captures shape, which is assumed stable.

Relation to Prior Work

Three strands motivate the problem. Garg et al. showed that dynamic scoring, meaning standardizing reconstruction errors by a sliding-window mean and variance (Gauss-D), matters more than detector architecture on industrial benchmarks (An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series). Their window, however, does not separate noise-floor growth from anomaly contamination. Griffin sets its threshold from a moving average of errors on recently processed normal instances (Griffin). That is exactly the censored update whose bias we study. AlerTiger widens bands with an IQR of week-over-week differences and drops the latest three days to avoid absorbing slow anomalies (AlerTiger). This is a robust heuristic with no model of how the noise was generated. We differ from all three by treating the exclusion rule as a known truncation and correcting for it, rather than choosing between poisoning and ad hoc omission.

The tail-modeling component comes from SPOT and DSPOT (Anomaly Detection in Streams with Extreme Value Theory). DSPOT subtracts a moving average and fits a Generalized Pareto tail to the residuals, which implicitly assumes the residual scale is stable between updates. We add a corrected time-varying scale in front of the tail fit, so the Generalized Pareto parameters describe tail shape rather than a mixture of noise regimes. The oddwater procedure fits an exponential tail to typical scores for sensor outliers (A Feature‐Based Procedure for Detecting Technical Outliers in Water‐Quality Data From In Situ Sensors) and faces the same scale-drift issue.

Two lines of work constrain the design. The truncated-regression guarantees hold only for Gaussian noise, and sample complexity grows sharply as the probability of surviving truncation falls. Our truncation is mild, since most points survive, so the complexity problem is small. The Gaussian assumption, though, is a real limitation, and we test it directly. Where noise depends on signal level, a variance-stabilizing transform can make the bulk closer to homoscedastic. ConvexVST shows that an optimized transform beats closed-form Anscombe-type transforms under clipping by nearly 2 dB PSNR on fluorescence microscopy (ConvexVST). ConvexVST needs sensor calibration data, however, so we use it only as an optional preprocessing step in settings where such data exist. UNSURE estimates an unknown noise level jointly with a denoiser through a zero-expected-divergence constraint (UNSURE). We considered porting it and rejected it for the core method: it estimates a global level during training, whereas the thresholding problem needs a cheap per-window estimate. Variance-stabilized density estimation regularizes the variance of log-likelihood on static tabular data (Anomaly Detection with Variance Stabilized Density Estimation). It works at the scoring stage, not the thresholding stage, and does not address streaming.

Research Program

The work has three stages. The first is analytical. We derive the fixed point of the iterated censored-variance map for Gaussian and Student-t bulks as a function of the censoring multiple $k$ and the window length. We then translate the resulting scale bias into realized false-alarm rate at target risk $q$. This yields a predicted miscalibration curve that the experiments must match.

The second stage is algorithmic. We implement the streaming estimator: truncated maximum likelihood over the accepted residuals in each window, using the exactly known truncation set. That set includes the adaptive threshold, the two-sided sensor clipping limits, and any segment-exclusion rule. The estimator is warm-started, runs a few gradient steps per update, and feeds standardized scores into a SPOT-style tail. For multivariate data we apply the estimator per channel to the residuals of a univariate autoencoder, following Garg et al.'s finding that this simple detector is competitive.

The third stage is evaluation. The evaluation is organized around calibration, not ranking. The TSB-AD authors point out that rank metrics such as Volume Under the Surface of the precision-recall curve (VUS-PR) hide whether a usable operating threshold exists (The Elephant in the Room). We therefore report, as primary outcomes, the realized false-alarm rate on normal stretches relative to target $q$, and event-level detection measured by the composite F-score of Garg et al. That score is the harmonic mean of time-wise precision and event-wise recall, and it avoids the point-adjustment inflation both papers criticize.

Baselines

The naive reference is a fixed threshold calibrated once on an initial window, with no updates. It measures how much any adaptation is worth under noise-floor drift. The second reference is an uncensored rolling mean and standard deviation, which is Gauss-D as in Garg et al. It isolates poisoning. The third is naive censored updating, in which the window keeps only unflagged points and uses their sample variance, as in Griffin's self-thresholding. This is the configuration the proposal claims is biased, and it serves as the key ablation. Every method shares the same univariate-autoencoder residuals, window length, and warm-up period, and each gets the same hyperparameter grid for window length and censoring multiple, tuned on held-out normal data only.

The strongest existing thresholding method is DSPOT from Siffer et al., run on the same residuals with its recommended initial-threshold quantile. For DSPOT we have one reported figure on the network SYN-flood data (MAWI): SPOT reached "a True Positive Rate of 86% with a False Positive Rate under 4%" (Anomaly Detection in Streams with Extreme Value Theory). We will reproduce this setting to check our implementation. We found no reported composite-F-score or false-alarm-calibration result on Lacuna for DSPOT on TSB-AD, SMD, SWaT or MSL, so those numbers will be produced here under identical conditions.

We also include a robust-scale baseline: rolling median with median absolute deviation, plus the same Generalized Pareto tail. A second robust baseline is AlerTiger's IQR-based irregularity scaling without its supervised stage. Finally, a deliberately simple correction serves as an ablation: the censored sample variance multiplied by the closed-form Gaussian truncation factor for multiple $k$. If this constant-factor fix matches full truncated maximum likelihood, we will report that the correction, not the optimizer, is what matters.

Beating the baselines means two things. The first is realized false-alarm rates within a factor of 1.5 of target $q$ across injected noise regimes, where naive censoring and DSPOT drift outside that range. The second is a composite F-score at least as high as DSPOT's, with no loss of detection bought by looser thresholds.

Why not just…?

Why not just use a robust scale estimator like the median absolute deviation instead of censoring? This is the most serious competitor. The median absolute deviation tolerates contamination up to half the window without excluding anything, so it never truncates. Its weaknesses are statistical efficiency and shape assumptions. Under Gaussian noise it uses the data far less efficiently than maximum likelihood, so thresholds from short windows are noisier. Its conversion to a standard deviation also assumes a symmetric Gaussian bulk, which fails under the one-sided clipping that sensor data show. Neither weakness guarantees a practical loss, so the experiments compare the two directly across window lengths. If the median absolute deviation matches our calibration at the window lengths operators use, we will say so, and the proposal's value then reduces to diagnosing the censoring bias in existing systems such as Griffin.

Why not just multiply the censored variance by a known correction constant? For a symmetric Gaussian bulk with a fixed cutoff multiple, truncated maximum likelihood essentially reduces to this. We therefore run it as an explicit ablation rather than dismissing it. The full estimator should matter only when the truncation set is irregular: one-sided exclusion, clipping at the sensor's range limits, or a cutoff that moves with the adaptive threshold. The water-quality and injected-clipping experiments are designed to show whether those cases occur often enough to justify the extra machinery.

Why not just widen the window, or omit recent data as AlerTiger does? Longer windows reduce the spread of the estimate but lag behind real changes in the noise floor. Omitting recent data protects against slow anomalies but not against censoring bias, which remains even in stationary noise. The step-change experiments measure lag directly: we record the time each method takes to recalibrate after a known noise-floor jump.

Proposed Experiments

The decisive experiment uses normal stretches of the univariate TSB-AD collection, TSB-AD-U (The Elephant in the Room). Into these we inject controlled noise-floor regimes of three kinds: step changes in scale, slow ramps, and signal-dependent scaling. We add synthetic spikes and level shifts of known amplitude. Because the true noise scale is known, we can measure each method's scale-estimation error and realized false-alarm rate against the analytically predicted miscalibration curve. The first claim is confirmed if naive censoring settles at the predicted biased fixed point and our estimator removes the bias. It is falsified if the bias is negligible at practical censoring multiples.

We then move to real streams without injection. These are MAWI network traffic from the SPOT paper; SMD, SWaT and MSL from Garg et al.; and the Sandy Creek and Pioneer River water-quality series, where sensor clipping and bursty technical outliers provide naturally irregular truncation. On these we report the composite F-score and false-alarm calibration, and VUS-PR only as a secondary check that standardization does not hurt ranking.

A final stress test replaces Gaussian noise with Student-t noise at several degrees of freedom. This probes the Gaussian limit of the truncated estimator. We will report where calibration breaks down, and whether a truncated Student-t likelihood recovers it, instead of claiming robustness we have not shown.