Deciding Later on Purpose: Streaming Neyman-Pearson Rules with a Watch Option

Fraud detectors, grid monitors and heavy-rain alarms all face the same bind. They must act the instant an event arrives, false alarms and misses cost very different amounts, and the true label shows up hours or days later. Standard practice is to train one scoring model and pick a single threshold. That choice quietly assumes the scores of the two classes are well separated, so a single cut is enough. In real streams the classes overlap. There is a band of scores where the model at arrival time cannot tell the classes apart, however it is trained. Yet the stream often resolves this band on its own if you wait. In card fraud, 70% of compromised cards transact again within 24 hours, while only 1% are blocked within the first hour. A model that can see those later transactions nearly doubles early detection (The Importance of Future Information in Credit Card Fraud Detection). That model can never block anything at arrival, so its authors confine it to a human review queue.

We propose a three-way decision rule for live streams. At arrival, each event is blocked, allowed, or placed on watch. A watched event is resolved later by a second model that sees what happened next. All three thresholds are chosen together, online, so that the combined false-alarm rate on legitimate traffic stays below a fixed budget $\alpha$ while missed critical events plus the harm accrued during the watch period is minimized. In two sentences: when the classes overlap at decision time, the right response to asymmetric costs is to defer the ambiguous cases to a point where more evidence exists, not to move one threshold. We make that deferral a controlled, budgeted part of the decision rule rather than an afterthought.

Core Claim

The Neyman-Pearson formulation (minimize misses on the critical class subject to a hard cap $\alpha$ on false alarms) is the natural way to express asymmetric costs without guessing a cost matrix. Existing Neyman-Pearson methods are batch procedures. Neyman-Pearson Classification under Both Null and Alternative Distributions Shift and Transfer Neyman-Pearson Algorithm for Outlier Detection calibrate a threshold on a labeled target batch, and both return one two-way decision. Our claim has two parts. The first is conceptual. Under exact separability of the arrival-time score, a watch option has zero value, because a single threshold already achieves zero misses at any $\alpha$. The value of deferral grows with the overlap between the classes' arrival-time scores, and it is bounded by how much the future-informed score reduces that overlap. The separability assumption behind margin-based theory, from Margin Maximizing Loss Functions onward, is exactly what hides the need for deferral. The second part is practical. The three thresholds can be kept calibrated on a drifting stream with delayed labels, keeping realized false-alarm rates near $\alpha$ at per-event cost small enough for real-time use.

Let $s(x)$ be a score computed only from the past, and $r(x)$ a score computed after $m_F$ subsequent events. The rule blocks if $s > \tau_b$, watches if $\tau_w < s \le \tau_b$ (and then blocks if $r > \tau_r$), and allows otherwise. With $Y=1$ the critical class, $W$ the watch event, and $E$ the harm realized while watching (for example, further fraudulent spend on the same card), the program is

$$\min_{\tau_b,\tau_w,\tau_r}\; P(\text{allow}\mid Y=1) + \lambda\, \mathbb{E}[E \cdot \mathbf{1}_W \mid Y=1] \quad \text{s.t.}\quad P(\text{block at either stage}\mid Y=0)\le\alpha,$$

where $\lambda$ converts watch-period harm into the same units as a miss. The rule falls back to an ordinary Neyman-Pearson threshold when $\tau_w=\tau_b$, and to "allow everything, review later" when $\tau_b=\infty$. Both extremes are therefore baselines inside the same family.

Research Program

The first piece is analysis. For fixed score distributions, we will characterize the optimal watch band as a function of three quantities: the overlap of the class-conditional distributions of $s$, the gain in separation from $s$ to $r$, and the exposure cost $\lambda$. The goal is a clean statement of when deferral strictly helps. We expect the Type-II gain to vanish as overlap in $s$ vanishes, and to be capped by the Neyman-Pearson Type-II error of $r$ at the same $\alpha$. This part turns the separability direction from a background assumption into a quantity we measure and vary.

The second piece is online calibration. It works because the constrained class is also the abundant one. Legitimate traffic makes up more than 99.9% of card transactions, so false-alarm quantiles can be estimated from unlabeled traffic in a sliding window. The inflation caused by unseen fraud in that window is bounded by the fraud base rate. Delayed labels, as they arrive, correct that contamination and are used to re-estimate the miss side. For pending labels we reuse the forgetting-and-simulating idea from Stream-based active learning for sliding windows under the influence of verification latency. That paper drops window samples whose labels will arrive too late and fills in-flight labels with bootstrapped or fractional weights. We adapt it to stochastic rather than known delays, which removes its known-latency assumption. The watch band is re-optimized on a slower clock. When drift detection fires, we keep restore points in the manner of AiRStream (Analyzing and repairing concept drift adaptation in data stream classification), so a false drift alarm cannot permanently distort the thresholds.

The third piece is the models themselves. They are deliberately ordinary: a causal LSTM or gradient-boosted scorer for $s$ and a bidirectional model for $r$, following the setup in the future-information paper. The contribution is the decision layer, not the architecture.

Baselines

The naive reference point is a cost-weighted classifier with one threshold fixed on a held-out offline split, then deployed unchanged. This is the static reweighting practice that direction 3 papers use (Chakraborty et al.; Hu et al.). It shows how badly false-alarm control degrades under drift.

The strongest competitor for false-alarm control is a streaming two-way Neyman-Pearson rule. It uses the same causal score $s$, with a threshold set by the rolling unlabeled-quantile method with delayed-label correction described above. In other words, it is our system with the watch option removed. This is the obvious streaming adaptation of the batch methods of Kalan et al., and it is the fairest possible comparison. We also run the batch CP-Solver of Neyman-Pearson Classification under Both Null and Alternative Distributions Shift and TLNP, re-fit periodically on the most recent labeled window. On the ClimSim heavy-rain clusters, TLNP reports Type-II error 0.26 versus 0.30 for target-only. On NASA POWER both reach 0.37. Those are batch numbers in a transfer setting, so they locate the difficulty rather than serve as a directly comparable target.

The strongest competitor for detection power is the bidirectional model applied after the fact to all traffic, the operating mode of The Importance of Future Information in Credit Card Fraud Detection. That paper reports transaction AUCPR 0.403 for the Bi-LSTM versus 0.344 for the causal LSTM, and early AUCPR 0.254 versus 0.165 and 0.110 for random forest. It sets an upper reference for detection quality but pays full exposure on every fraud.

The key ablation is a fixed heuristic watch band: watch everything whose causal score falls between two quantiles chosen by grid search offline, with no joint Neyman-Pearson optimization and no online recalibration. This isolates whether the gain comes from our allocation and calibration or merely from having a second stage at all.

All methods share the same scoring models, the same streams replayed strictly in time order with realistic label delays, and the same hyperparameter search budget. Beating the field means two things together. The realized false-alarm rate must stay within a small tolerance of $\alpha$ in at least as many sliding windows as the two-way streaming rule. And the combined objective (misses plus weighted exposure) must be lower than the two-way rule, the allow-all-then-review mode, and the heuristic band, across a range of $\lambda$.

Why not just…?

Why not just keep a rolling quantile threshold on the causal score? For controlling false alarms, this is a real threat, and we adopt it as our calibration component rather than claim it as new. What it cannot do is lower misses in the overlap region. At a fixed $\alpha$, no single threshold on $s$ beats the Neyman-Pearson Type-II floor set by the overlap of $s$. The synthetic overlap sweep tests this directly: if the watch option gives no gain once overlap is large and $r$ separates better, the proposal fails.

Why not just send ambiguous cases to human review, as learned abstention does? Learning to Abstain in the Presence of Uninformative Data trains a selector that abstains on uninformative inputs and reports selective risk around 11–20% versus 20–45% for SelectiveNet-style baselines. Its guarantees assume informative and uninformative inputs have disjoint supports, and it is a static batch method with no model of what waiting reveals. Our watch option is abstention that resolves itself from incoming data at no annotation cost. We will still run a selector-plus-human-queue variant with the same review budget, to see whether human resolution at a fixed capacity beats automated resolution.

Why not just run the bidirectional model on everything and skip real-time blocking? This recovers most detection power but lets every fraud run through the whole delay. The file's own statistic that 70% of compromised cards transact again within a day means this mode incurs exactly the exposure $E$ our objective penalizes. Our experiments report exposure directly, so the size of this gap is measured rather than assumed.

Why not just add a second threshold by hand? This is the heuristic-band ablation. If it matches the full method under drift, the contribution shrinks to the analysis and the calibration. We will report that honestly.

Proposed Experiments

The decisive synthetic study builds card-like event streams with an explicit overlap knob for the causal score and a separate knob for how much future events reveal. It also includes injected drift and stochastic, bursty label delays. The predicted signature is specific. The watch option's Type-II gain should rise with overlap, stay near zero under separability, and saturate at the Type-II floor of $r$. We also expect realized false-alarm rates to stay near $\alpha$ only when the delayed-label correction is on.

On real data we use four sources. The first is the European credit card dataset of 284,807 transactions. It lacks card identifiers, so there "future context" means the next events in time rather than the same card, which is a weaker test. We note that CS-SVM's 98% accuracy there came from a 5,094-transaction static subsample (Application of Credit Card Fraud Detection Based on CS-SVM) and is not comparable to streaming Neyman-Pearson metrics. The second is the NASA POWER and ClimSim heavy-rain settings of the Neyman-Pearson transfer papers, where the natural watch action is waiting a few hours for more observations before issuing an alert. The third is the Electricity, NOAA Weather and Rialto streams from the verification-latency study, with minority-class Neyman-Pearson constraints imposed. The fourth, if access can be arranged, is the industrial 30-million-transaction dataset of the future-information paper, which is the only one with true card sequences.

The metrics are the fraction of sliding windows in which the realized false-alarm rate exceeds $\alpha$, Type-II error, exposure-weighted miss cost, watch rate, and per-event decision time. The single most informative check is the curve of Type-II gain against measured score overlap across all datasets. If gains track overlap as the analysis predicts, the case for deferral is made. If they do not, the watch option is a heuristic and the proposal should say so.