Detectors for rare events in continuous streams, such as seizure subtypes in EEG, rare fault types in sensor data, or uncommon clinical deteriorations, are usually trained with tricks that boost the rare classes and then evaluated on data shuffled across time. Two separate literatures have shown that each habit inflates reported performance. Randomly mixing past and future data hides the decay a model suffers after deployment. Scoring with overlap or aggregate curve metrics hides whether an alarm arrived on time. This note argues that the two failures are linked, and that the link runs through one small component: the operating point. By operating point we mean the per-class decision thresholds and prior-correction offsets that long-tail methods fit to the class frequencies seen during development.
Our core claim is that much of the performance a rare-event detector loses between random and chronological evaluation is lost at the operating point, not in the learned representation. Thresholds tuned to a past prevalence produce late onsets during bursts of a rare class and scattered false alarms during quiet periods. The metric most used to study temporal decay, the area under the ROC curve (AUROC), is unchanged by any choice of threshold, so it is structurally blind to this error. Metrics that score alarms by their distance in seconds from true event boundaries do see it. If the claim holds, a large share of chronological decay can be repaired without retraining, by refreshing thresholds causally from recent past labels and choosing them with a distance-aware objective. We will test the claim directly, and the test is designed so that it can fail.
Background and Gap
The chronological-evaluation line of work supplies the motivation. Evaluating Model Performance in Medical Datasets Over Time (EMDOT) introduced rolling backtests: the model is retrained at successive cutoff dates and tested only on later data, across SEER, MIMIC-IV, COVID-19 and transplant records. The paper found that random splits are optimistic and that complex models go stale faster than simple ones. All of its evidence is AUROC and prevalence. It therefore cannot distinguish a model whose ranking of cases has decayed from one whose ranking is intact but whose thresholds no longer fit the data. Its own remedy, sliding-window retraining, has a known cost: it requires retraining at every cutoff, and in rare-event domains short windows can contain almost no tail examples. The leakage literature adds a further constraint. Global normalization and full-series decomposition leak future statistics into past features (Horn et al.; Yang), so every preprocessing step in our protocol must use past data only. We extend this concern to a leakage channel the file does not name: validation data drawn across time leaks the future class prior into threshold selection.
The distance-aware evaluation line supplies our measuring tools. Local Evaluation of Time Series Anomaly Detection Algorithms defines affiliation metrics. These measure, in real time units, how far each prediction is from its nearest true event, normalized so that 0.5 equals random guessing. The paper shows that fragmented adversarial alarms that score near 0.99 on range metrics fall to about 0.50 under affiliation. Towards a Rigorous Evaluation of Time-Series Anomaly Detection shows that point adjustment, which credits a whole anomalous segment if any single point inside it is flagged, lets random scores reach an F1 of 0.969 on SWaT. It proposes PA%K as a stricter variant. Real-Time Seizure Detection using EEG introduces MARGIN, which requires alarms to fall within 3–5 second tolerance buffers around onset and offset. All three are binary and are evaluated on static or cross-sectional splits.
The long-tail line supplies the methods whose operating points we study. Seesaw Loss rescales gradients using cumulative class counts from the training set. Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation balances class-level and transition-level bias and shows that focal loss and logit adjustment cause flickering segments. Addressing Mark Imbalance in Integration-free Marked Temporal Point Processes learns class-specific thresholds after predicting the event type first. Each of these fixes a class prior or threshold once, on a static split. None asks what happens to that prior when the stream moves forward in time.
The closest single pieces are EMDOT, which covers chronological decay but only with a threshold-blind metric, and the MARGIN study, which covers distance-aware scoring on rare events but only on static splits. Neither separates decay into a representation part and an operating-point part, which is our central measurement.
Core Claim and Decomposition
Fix a backtest period $k$ and a rare class $c$. Let $U$ be a distance-aware utility, primarily the affiliation F1 for class $c$ on period $k$, with onset latency reported as well. Let $\theta_{\text{dev}}$ be the threshold and offset chosen during development, and let $\theta^\ast_k$ be the oracle threshold chosen on period $k$ itself. The oracle is not deployable; it serves only as a measuring instrument. We split the random-to-chronological gap as follows:
$$ \underbrace{U_{\text{rand}} - U_k(\theta_{\text{dev}})}_{\text{total gap}} \;=\; \underbrace{U_{\text{rand}} - U_k(\theta^\ast_k)}_{\text{representation gap}} \;+\; \underbrace{U_k(\theta^\ast_k) - U_k(\theta_{\text{dev}})}_{\text{operating-point gap}} $$
Here $U_{\text{rand}}$ is the utility on a random split. Under AUROC the second term is zero by construction, which is exactly why EMDOT-style evaluation cannot see it. We predict three things. First, the operating-point gap is a large fraction of the total gap for tail classes and a small fraction for head classes. Second, it grows with the drift in class prevalence between development and period $k$. Third, stronger long-tail compensation, such as Seesaw factors or logit adjustment, makes it worse during quiet periods, because boosting a class that was rare but bursty produces off-event false alarms.
Research Program
The first component is a leakage-clean rolling backtest built on the EMDOT package. Every normalization, imputation and threshold is fit only on data before the cutoff. A temporal buffer separates train and test, following the ENSO practice in Temporal Convolutional Networks for the Advance Prediction of ENSO. We also run a deliberately leaky variant, with thresholds selected on a validation set shuffled across time, to measure how much optimism threshold selection alone contributes.
The second component extends affiliation metrics to multiple classes. The extension builds affiliation zones for each class separately, but this creates a problem we must handle openly. A rare class has few events, so its zones are long, and the random-guess normalization inside those zones becomes lenient. We therefore report affiliation alongside onset latency in seconds and MARGIN at 5 seconds. The claim is accepted only if the conclusions agree across all three.
The third component is the remedy, causal operating-point refresh. At each cutoff, the backbone stays frozen. Per-class thresholds and logit offsets are re-fit on a trailing window of labeled past data. The selection objective is the distance-aware utility, not frame F1. For classes with few events in the window, the estimate is shrunk toward the all-history value, weighted by the number of events in the window. This directly addresses the problem that short windows can hold almost no tail examples. The refresh costs a threshold search rather than a retraining run, which also addresses EMDOT's stated computational burden.
Baselines
The naive reference is a single model trained before the first cutoff. It uses a threshold chosen on a validation set shuffled across time, which is today's common practice, and is never updated. Every other result is reported relative to it.
The static long-tail baselines are focal loss, logit adjustment from training priors, and Seesaw-style gradient rebalancing, each with its threshold fixed at development time. For the point-event dataset we add the learned class-specific thresholds of IFNMTPP. On USearthquake, IFNMTPP reports a Macro-F1 of 0.1382 on a static split. We will reproduce that figure on the static split before running it chronologically, so that any drop is attributable to time rather than to our reimplementation. For the EEG backbone, Lee et al. report AUROC 0.92 and AUPRC 0.91 for ResNet-short with an LSTM on bipolar TUSZ, on a static patient split. No reported MARGIN number was found on Lacuna for any model, so our MARGIN numbers will be the first chronological ones rather than a comparison.
The strongest competitor is EMDOT sliding-window retraining with the same long-tail loss. It adapts both the representation and the operating point, and it is the remedy the chronological literature already recommends. For a fair comparison, both arms use the same backbones, the same cutoffs and the same trailing window. Hyperparameter search budgets are matched in trials, and both arms have identical access to past labels. Our refresh beats retraining if it recovers at least as much of the tail-class distance-aware utility at a small fraction of the compute. Matching it at lower cost is a positive result. Losing to it only at head classes is consistent with our claim.
The key ablation is refresh selected by frame F1 versus refresh selected by distance-aware utility, with shrinkage and window length held fixed. A second ablation removes the shrinkage. The oracle $\theta^\ast_k$ serves as the upper bound on everything threshold-based.
Why not just…?
Why not just retrain on a sliding window, as EMDOT does? This is a real threat, and we test it head to head. Our prediction is that retraining is costly and, for tail classes, often worse than refreshing thresholds: short windows starve the model of rare examples, which the file flags as the data scarcity versus recency dilemma. If retraining dominates at tail classes even after accounting for compute, then the representation gap is what matters, and our claim is weakened.
Why not just apply logit adjustment with priors re-estimated from recent data? This is essentially one arm of our remedy, and the answer may be that it suffices. The ablation settles whether choosing thresholds by distance-aware utility adds anything beyond prior tracking. Earlier evidence suggests it might: prior correction applied frame by frame causes flickering, which Pang et al. documented, and flickering is exactly what affiliation precision penalizes. If rolling logit adjustment ties with our method, we will report the simpler method as the recommendation. The decomposition result stands either way.
Why not just report AUPRC on a chronological split? AUPRC does depend on prevalence, but it weights every positive frame equally. It therefore rewards catching the middle of a long seizure while missing its onset. Lee et al. show a model with high AUPRC can still have poor clinical timing. We report AUPRC anyway and test whether its ranking of methods across periods agrees with the ranking by onset latency.
Why not just use point adjustment or PA%K? Point adjustment is ruled out by Kim et al., who show random scores reach an F1 of 0.969 on SWaT. PA%K is reasonable, and we include it as a secondary metric, but its fixed $K$ does not measure lateness in seconds, which is the error we expect stale thresholds to cause.
Proposed Experiments
The primary testbed is TUSZ multi-class seizure-type detection, a long-tailed set of subtypes, ordered chronologically by recording session. Whether the corpus metadata supports true calendar ordering must be verified first. If it does not, we fall back to within-patient session order and state that limitation. The second testbed is MIMIC-IV onset-style prediction through the EMDOT anchor-year groups. The third, USearthquake, is a point-event control with drifting mark frequencies. The metrics are per-class affiliation F1, onset latency, MARGIN at 5 seconds, PA%K and AUROC.
The decisive check is the decomposition itself. If oracle re-thresholding recovers less than a third of the chronological gap in tail-class affiliation F1 on TUSZ and MIMIC-IV, the operating point is not where the optimism hides, and we abandon the refresh method in favour of reporting that negative result. The second check is whether the size of the operating-point gap correlates with period-to-period prevalence drift, measured as total variation distance between class frequencies. The third is whether the leaky, time-shuffled threshold selection explains a measurable share of the random-split optimism that EMDOT reports.
References
- Evaluating Model Performance in Medical Datasets Over Time
- Set Functions for Time Series
- A Hybrid Pork Price Forecast Model Based on LightGBM and Segment-wise Attention
- Local Evaluation of Time Series Anomaly Detection Algorithms
- Towards a Rigorous Evaluation of Time-Series Anomaly Detection
- Real-Time Seizure Detection using EEG: A Comprehensive Comparison of Recent Approaches under a Realistic Setting
- Seesaw Loss for Long-Tailed Instance Segmentation
- Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation
- Addressing Mark Imbalance in Integration-free Marked Temporal Point Processes
- Temporal Convolutional Networks for the Advance Prediction of ENSO