Scoring Event Detectors by When They Speak, Not Where They Point

Many systems that detect events in time, such as seizure alarms, sepsis warnings, and industrial anomaly monitors, are scored offline. The evaluator compares the predicted start and end of each event with the labeled interval, often allowing a small "tolerance window" so that a prediction one second off is not counted as both a miss and a false alarm. Offline scoring checks where a predicted boundary sits. It does not check when the system could have known that boundary. A detector built with a bidirectional network, a centered smoothing filter, whole-recording normalization, or post-processing that merges segments only after an event has ended can place its onset exactly. In deployment, that same system could not commit to the onset until seconds or hours later. Tolerance windows were designed to forgive harmless misalignment, and they make this hidden use of future information even harder to see, because a late commitment and a slightly misplaced boundary look identical.

We propose emission-time scoring. Every detector must produce a time-stamped log recording the moment it commits to each boundary. A distance-aware interval metric then scores those committed boundaries, with an asymmetric cost for lateness. The difference between a detector's offline score and its emission-time score is a direct measure of how much its reported performance depends on look-ahead. We call this difference the look-ahead gap. We will measure it on chronologically split versions of seizure, sepsis, and industrial anomaly benchmarks. We expect it to be large, to grow with tolerance width, and to reorder method rankings that offline benchmarks currently report.

Core Claim

Three bodies of work each handle one piece of this problem, and none handles the conjunction. The first is relaxed interval metrics. Local Evaluation of Time Series Anomaly Detection Algorithms defines "affiliation" metrics, which score detections by average temporal distance to nearby labeled events, normalized so that 0.5 is chance. Temporal Knowledge Base Completion introduces aeIOU, an overlap score that still gives partial credit to disjoint intervals that are close together. Time to Transfer adds slack windows to its handoff metric. All of these score boundary position only.

The second body is streaming clinical evaluation. Real-Time Seizure Detection using EEG runs models on sliding windows and scores alarms with MARGIN, a 3–5 second tolerance around onset and offset. The PhysioNet 2019 sepsis challenge uses a per-hour utility function that rewards warnings up to 6 hours early and penalizes late or premature ones. These efforts respect causality, but each is tied to one task. MARGIN's window is symmetric, and the sepsis utility is defined per time step rather than per interval.

The third body is temporal-leakage studies, which motivate the problem. Aeolus finds that random splits inflate AUC by an average of 0.057. EMDOT finds that random splits give optimistic results on six clinical datasets. Horn et al. ( Set Functions for Time Series) show that normalization or smoothing computed over a whole dataset leaks future statistics. These studies look at leakage between the training and test splits, and they score with point metrics such as AUC. None of them examines look-ahead inside the test stream of an interval detector.

Our claim is that, for interval detection, there is a leakage channel separate from split leakage. It is look-ahead at inference time, and position-based interval metrics cannot see it, whatever their tolerance. The channel becomes measurable once the scorer charges for commitment time. The key ingredient is separating a boundary's position $\hat{s}$ from its emission time $e \ge \hat{s}$, which is the earliest moment the detector's log contains it. For a labeled onset $s$, emission-time credit is

$$c = \underbrace{A(\hat{s}, s)}_{\text{position}} \cdot \underbrace{u(e - s)}_{\text{commitment}},$$

where $A$ is the affiliation-style distance credit in $[0,1]$. The function $u$ is a non-increasing utility of lateness: it equals 1 for $e \le s + \tau$ and decays to 0 by $e = s + L$. Here $\tau$ is the clinically allowed delay and $L$ is the point past which an alarm is useless; their values come from the task, for example the 6-hour sepsis window. Offsets and false alarms are treated the same way. The look-ahead gap at tolerance width $\Delta$ is

$$G(\Delta) = M_{\text{offline}}(\Delta) - M_{\text{emit}}(\Delta),$$

where $M$ is the aggregate metric computed with or without the commitment term.

Research Program

The first deliverable is the protocol. Detectors are wrapped in a harness that feeds test streams forward in time. The harness records every change to the detector's current set of predicted intervals along with the wall-stream timestamp. A boundary that is later revised counts only from its final committed emission. Revisions themselves are scored as false alarms if they retract an alarm.

For methods that are inherently offline, such as bidirectional models or segment-merging post-processing like the hysteresis merging in How to Count Coughs, the harness gives the method a growing prefix of the stream. It re-runs the method at a fixed stride and logs the emissions that the method would have produced on that prefix. This makes any detector scorable without re-engineering it, and it is the main practical difference from simply "running things in streaming mode."

The second deliverable is a controlled validity study. We take causal detectors and add known amounts of look-ahead $k$: centered smoothing of width $k$, normalization statistics computed over a future window of length $k$, or a bidirectional context of $k$ seconds. A valid leakage-sensitive metric should leave $M_{\text{emit}}$ flat or decreasing in $k$, while $M_{\text{offline}}$ rises. If $M_{\text{emit}}$ also rises with injected look-ahead, the scoring rule is broken and we revise $u$ before going further.

The third deliverable is the audit. We will measure $G(\Delta)$ for families of real detectors under chronological splits and test whether rankings change. We will also test whether drift between training and test periods amplifies the gap. Horn et al.'s results suggest that whole-recording normalization becomes more helpful offline exactly when the test distribution has moved away from training. That interaction between split chronology and in-stream look-ahead is untested.

Baselines

The naive reference is the offline metric each benchmark already uses, run on the same chronological split. For TUSZ this means frame-level AUROC/AUPRC plus symmetric MARGIN. For SWaT it means affiliation precision and recall. For PhysioNet it means the official utility computed on predictions generated with full-record preprocessing. The reported numbers set the scale. ResNet-short + LSTM reaches AUROC 0.92 and AUPRC 0.91 on bipolar TUSZ (Lee et al.). Winning sepsis-challenge teams scored utility between 0.33 and 0.36 (Reyna et al.). No reported MARGIN values were found on Lacuna for TUSZ, and no per-detector affiliation scores were found for iForest or seq2seq on SWaT, so we will compute both ourselves.

The strongest existing competitor is streaming evaluation as practiced by Lee et al.: causal sliding windows scored with MARGIN. That protocol already enforces causality for the models it was applied to. We will run it on exactly the same data, splits, and detector checkpoints as our protocol. The test is whether emission-time scoring exposes inflation that streaming-plus-MARGIN misses. Two sources are candidates: post-processing applied after the stream, and late commitments that fall inside MARGIN's symmetric window. Beating this competitor means finding detector pairs whose MARGIN ordering contradicts their emission-time ordering, and showing on the injected-look-ahead study that only the emission-time score tracks the true amount of injected future information.

A second competitor is the sepsis utility function itself, rescored per interval. It already contains an asymmetric lateness term, so it is the closest prior scoring rule. The ablation that isolates our key ingredient is a position-only asymmetric metric. It uses the same $u$ but applies it to $\hat{s} - s$ instead of $e - s$. If this ablation detects the injected look-ahead as well as emission time does, then separating position from emission time is unnecessary and the contribution shrinks to "use asymmetric tolerance." All variants receive the same hyperparameter budget per detector, tuned on a chronologically earlier validation window.

Why not just…?

Why not just run every model in causal streaming mode, as Lee et al. do? For models designed to be causal, this is close to what we propose, and we say so. It falls short in two places. First, many published detectors and post-processing steps cannot be run causally without being rewritten. Second, causality of the network says nothing about post-hoc merging or about late alarms that a symmetric window forgives. Our harness turns any detector into an emission log, and the head-to-head with streaming-plus-MARGIN will show whether the extra machinery changes any conclusion. If it changes none, we will report that streaming-plus-MARGIN suffices.

Why not just make the tolerance window asymmetric, crediting only early or on-time boundaries? This is the most serious threat, and it is cheap. Its weakness is that a non-causal detector can place a boundary early while having committed to it late, so position-based asymmetry still credits look-ahead. The position-only ablation described above tests exactly this on injected look-ahead. If the gap between the ablation and emission-time scoring is negligible, we adopt the simpler rule.

Why not just fix leakage at the split level with chronological splits and embargo gaps? Chronological splits (EMDOT, Aeolus) and embargo buffers (Temporal Convolutional Networks for ENSO) stop the model from training on the future. They do not stop a test-time detector from reading ahead within the test recording, which is the channel at issue. We keep chronological splits throughout, so any gap we measure is on top of what split hygiene already removes.

Proposed Experiments

The datasets are the TUH EEG Seizure Corpus V1.5.2, PhysioNet 2019, and SWaT. For TUSZ, recordings are ordered by session date where metadata permits; otherwise they are split by patient, and we flag this as weaker chronology. For PhysioNet, hospitals A and B are used for development and hospital C is held out, with within-stay hourly streaming. SWaT is a single trace cut chronologically, with an embargo gap. On each dataset we evaluate matched pairs of detectors. Each pair consists of a causal and a non-causal variant of the same architecture, plus standard post-processing on and off. Example pairs are causal CNN+LSTM versus BiLSTM, and causal z-scoring versus whole-recording z-scoring.

The decisive checks are three. In the injected-look-ahead study, $M_{\text{offline}}$ should increase with $k$ while $M_{\text{emit}}$ does not, and the position-only asymmetric ablation should fail to separate the two. Across real detector pairs, $G(\Delta)$ should increase with tolerance width, measured on a sweep of $\Delta$ from 0 to 10 seconds on TUSZ and up to 6 hours on PhysioNet. At least one pair whose offline or MARGIN ranking favors the non-causal variant should reverse under emission-time scoring. Finally, comparing a random split against the chronological split tests whether drift enlarges $G$ for whole-recording normalization specifically.

We will report negative outcomes plainly. A small gap everywhere would mean current tolerance-based benchmarks are safer than feared, which is useful evidence for the field.