Anomaly monitoring systems rarely trust a single detector. They run a pool of them, such as nearest-neighbour, isolation-based and reconstruction-based models, often over many feature subsets, and raise an alarm when "the ensemble" finds something. Each extra detector, subset or configuration is one more place where a coincidental pattern can look significant. On a stream that never ends, this hidden search is a form of automated p-hacking. The statistical tools that control false alarms over time, called online false discovery rate (FDR) control, keep a running budget of allowed error and spend it test by test. Today they watch only one stream of evidence from one model. The ensemble tools that cope with "no single detector wins" either average scores heuristically or, as in sample-aware model selection, apply a static correction to each data point in isolation. Nobody accounts for the cost of searching across detectors inside the error budget that runs over time.
This note proposes accounting for that search by mixing evidence rather than by spending error budget. Each detector's score is turned into an e-value, a nonnegative statistic whose expected value is at most one when nothing anomalous is happening. At each time step the detectors' e-values are combined with weights chosen only from the past, and that single merged e-value is fed to an online FDR procedure that resists budget exhaustion. Because a weighted average of valid e-values with past-determined weights is still a valid e-value, whatever the correlation between detectors, the search over detectors costs no extra error budget. The open question is power: whether weights learned from past alarms can find the few detectors that matter quickly enough to beat a plain uniform average. That is the empirical bet of this proposal.
Core Claim
Let detector $j \in \{1,\dots,M\}$ produce an e-value $e_{t,j}$ at time $t$, obtained by calibrating its score against a held-out set of normal data. Let $w_{t,j} \ge 0$ be weights that sum to one and are computed from information available before time $t$. The merged evidence is
$$ E_t = \sum_{j=1}^{M} w_{t,j}\, e_{t,j}, \qquad \mathbb{E}\big[E_t \mid \text{past}\big] \le 1 \ \text{whenever each } \mathbb{E}\big[e_{t,j} \mid \text{past}\big] \le 1 . $$
The bound needs no assumption about how the $M$ detectors depend on one another, because it follows from linearity of expectation alone. That matters, since ensemble members trained on the same features are strongly correlated. It is exactly the property that per-sample Benjamini–Hochberg in ZODE does not have; ZODE runs a static correction across a model zoo for each test sample on its own. The stream $E_1, E_2, \dots$ is then given to an e-value online FDR rule from e-GAI, specifically its memory-decay variant mem-e-GAI. That paper proves online FDR control for e-values under arbitrary dependence, but only for one scoring model per time step.
The claim has three parts. First, validity is free: realized FDR stays at or below the target for any $M$, including pools that grow as an automated search adds feature subspaces. Second, post hoc selection is not free: picking "the detector that is working" from the stream and then testing with it inflates FDR. This is the oversearching failure described for rule search by the authors and for unadjusted exploration by Hullman et al.. Third, the only cost of searching is power. A uniform average divides a lone informative detector's evidence by $M$. Adaptive weights should recover most of that loss when few detectors are informative and $M$ is large.
Research Program
The weights follow a multiplicative update with a floor:
$$ w_{t+1,j} \propto (1-\gamma)\, w_{t,j}\, e_{t,j}^{\,\eta R_t} + \frac{\gamma}{M}. $$
Here $R_t \in \{0,1\}$ indicates whether time $t$ was declared a discovery, $\eta > 0$ is a learning rate, and $\gamma \in (0,1]$ is a floor that guarantees every detector at least $\gamma/M$ of the weight. Detectors that produced strong evidence on past alarms gain weight. The floor bounds the worst-case power loss against uniform mixing to a factor of $1-\gamma$. It also limits the self-reinforcing loop in which early discoveries lock the ensemble onto one anomaly type. That loop is a cousin of the "piggybacking" problem in online FDR, where wealth earned in a dense period licenses loose thresholds later.
The program has three components. The theoretical component makes the conditional-validity premise precise. Conformal e-values computed against a fixed calibration set are valid on average over that set, not conditionally on it. The work will either derive a calibration-conditional correction or use block-wise sample splitting, in which a calibration fold is refreshed only between weight updates, and prove that mem-e-GAI's guarantee carries over to the merged stream. The methodological component handles reference-set contamination. Following the anti-masking reservoir in SAFARI/ARES and the outlier filtering in CARE, calibration data are refreshed only from points the procedure did not flag. This tests whether memory decay's admitted precision cost, reported by Rebjock et al., lets anomalies leak into the calibration set. The empirical component, described below, measures when adaptive weighting earns its complexity.
The three input directions play different roles. Ensemble anomaly detection is the application. Online significance budgeting supplies the error-control engine: Rebjock et al. show that standard LORD and SAFFRON exhaust their budget when anomalies are rare, which motivates the decay variants, and e-GAI supplies the dependence-robust procedure. Spurious pattern discovery is the threat model and the evaluation lens, because the detector pool is a hypothesis search. The AWARE and QUDE line of work (Zhao et al., QUDE) charges each implicit test to the error budget. We charge the search to the mixture weights instead, because charging every detector at every time step would exhaust the budget almost immediately.
Baselines
The naive reference is the common practice: average PyOD-normalized scores across the pool (PyOD) and alarm above a fixed quantile chosen on calibration data. This reference has no error guarantee. It anchors what practitioners get today and shows how far realized false alarm rates drift on long streams.
The strongest single-stream competitor is decay-LORD or decay-SAFFRON from Rebjock et al.. It is run once on conformal p-values from one detector fixed in advance (Isolation Forest) and once on the in-hindsight best detector. The hindsight version is an oracle upper bound, not a fair method. No numerical result for these rules on the Server Machine Dataset (SMD) is reported on Lacuna. To beat this baseline, the merged ensemble must match the a priori detector's FDR and exceed its power on most streams, and must come close to the oracle's power without knowing which detector is best.
The strongest dependence-robust competitor is the uniform mixture, with $w_{t,j} = 1/M$, fed to mem-e-GAI. This is the ablation that isolates the proposal's key ingredient, adaptive weighting. If adaptive weights do not beat uniform at equal FDR, the contribution reduces to a validity result plus a recommendation to average e-values. A second ensemble competitor is a streaming adaptation of ZODE. It applies Benjamini–Hochberg across detector p-values at each time step, collapses the result to a single Bonferroni-style p-value, and passes that p-value to decay-SAFFRON. On the static image setting ZODE reduced average false positive rate on CIFAR-10 from 11.07% to 3.83% and on ImageNet-1K from 38.47% to 24.14% (ZODE). Our image-stream experiment uses ZODE's model zoo and out-of-distribution sets so that this baseline runs in its home setting.
A deliberately invalid baseline shows the cost of oversearching. It picks, every $k$ steps, the detector with the most alarms so far, using the same data that drives testing, and tests with that detector alone. All methods share detectors, calibration sets and target level. Hyperparameters such as $\eta$, $\gamma$ and decay rates get the same tuning grid, tuned on separate streams, never on the evaluation stream.
Why not just…?
Why not just average the detector scores and set a threshold? Averaging hides specialists: ZODE reports best-to-worst model gaps above 32%, and ADBench finds no single unsupervised method dominates across 57 datasets. A fixed threshold also has no control over time. Our experiments report realized FDR for this baseline over long streams rather than assuming it fails.
Why not just average the e-values uniformly? This is the real threat, and it is fully valid. Its weakness is predictable: a lone informative detector's evidence is divided by $M$, which bites when $M$ reaches the hundreds, as in automated subspace search. The pool-size sweep below settles the question. If uniform averaging matches adaptive weighting at every $M$ and sparsity level, we will report that and drop the weighting claim.
Why not just pick the best detector on calibration data and run one stream? Normal-only calibration data say little about which detector catches which anomaly type. Re-selecting from the stream is the oversearching that Hullman et al. measure at over 60% false conclusions in unadjusted exploration. The invalid-selection baseline quantifies the resulting FDR inflation directly.
Why not just use p-values with LORD or SAFFRON? Rebjock et al. show these exhaust their budget at low anomaly rates. e-GAI reports that p-value SAFFRON loses FDR control under AR(1) serial correlation, and correlation across detectors adds to that. We include decay-SAFFRON on Bonferroni-combined p-values to measure the power it gives up for its stronger assumptions.
Proposed Experiments
Validity is checked where ground truth is clean. ADBench tabular datasets (ADBench) are streamed in random order with injected local, global, dependency and clustered anomalies at rates down to 0.1%. A pool of PyOD detectors is crossed with random feature subspaces, giving $M$ from 5 to 500. We report realized FDR and decayed FDR against target, power, and the time until the effective threshold collapses.
The decisive check is a two-way sweep over $M$ and the fraction of informative detectors. Adaptive weighting must beat the uniform mixture in power at equal FDR when informative detectors are sparse and $M$ is large, and the invalid-selection baseline must exceed the FDR target. Duplicated-anomaly contamination, which ADBench links to a median −16.43% drop for unsupervised models, tests whether the refreshed calibration set resists masking under memory decay.
Real streams follow: SMD, as in Rebjock et al.; NAB including NYC taxi, where SAFARI reports a NAB score of 71.75 against 70.1 for HTM; and Yahoo S5, with no aggregate score reported on Lacuna. On these we report precision, recall and NAB score at matched alarm rates. No numerical e-GAI result on NYC taxi is reported on Lacuna, so we will rerun it ourselves. Finally, CIFAR-10 and ImageNet-1K out-of-distribution streams reuse ZODE's zoo to compare against its per-sample selection in an online setting.
The proposal fails if adaptive weights do not beat uniform mixing anywhere in the sweep, or if the conditional-validity correction costs more power than adaptivity gains.
References
- Enhancing the Power of OOD Detection via Sample-Aware Model Selection
- e-GAI: e-value-based Generalized $\alpha$-Investing for Online False Discovery Rate Control
- Oversearching and Layered Search in Empirical Learning
- Designing for Interactive Exploratory Data Analysis Requires Theories of Graphical Inference
- No Free Lunch But A Cheaper Supper: A General Framework for Streaming Anomaly Detection
- Sequential Ensemble Learning for Outlier Detection: A Bias-Variance Perspective
- Online false discovery rate control for anomaly detection in time series
- Controlling False Discoveries During Interactive Data Exploration
- Towards Quantifying Uncertainty in Data Analysis & Exploration
- PyOD: A Python Toolbox for Scalable Outlier Detection
- ADBench: Anomaly Detection Benchmark