Calibration Poisoning of Neyman-Pearson Detectors by Benign-Looking Data

Many safety-critical detectors fix their alarm threshold so that false alarms on benign data stay below a budget, say 5%, and then catch as many dangerous cases as possible. This is the Neyman-Pearson recipe. The threshold is almost always read off as an upper quantile of detector scores on a pool of benign examples, and in practice that pool comes from logs, crowdsourcing, or partner sites that nobody fully vets. We argue this calibration step is a weak point. An adversary who adds a few percent of entirely real-looking benign examples, all drawn from the upper tail of the benign score distribution, raises the threshold. The detector still keeps its false-alarm promise, so every monitored guarantee holds, yet it silently misses more attacks, and misses are the expensive error. Because the injected points are genuine benign values, global tests that compare the poisoned pool with benign data cannot tell them apart at realistic pool sizes.

The proposal has three parts. We characterize this attack, which we call tail mimicry. We show why it escapes standard checks: the threshold shift scales with the contamination fraction $\epsilon$ regardless of pool size, while a global two-sample test can only detect it once the pool holds roughly $1/\epsilon^2$ points. We then build a calibration rule that combines the large untrusted pool with a small trusted benign set. The trusted set is used only to audit the tail where the threshold lives. The cost asymmetry decides which direction of error the rule defends against. The central claim is that this tail-audited rule keeps the false-alarm guarantee while recovering most of the detection lost to poisoning. It should do so with clearly lower miss rates than either calibrating on the trusted set alone or applying the existing transfer method that treats the pool as an untrusted source.

Why the gap is real

Neyman-Pearson (NP) classification minimizes the miss rate (Type-II error) subject to a cap $\alpha$ on the false-alarm rate (Type-I error). Work on NP transfer learning has already shown that the benign, or "null", distribution is the fragile part. Neyman-Pearson Classification under Both Null and Alternative Distributions Shift shows that a shift in the null class invalidates the Type-I constraint. It handles this with a constraint-alignment step that recalibrates source benign data against target data. Transfer Neyman-Pearson Algorithm for Outlier Detection, called TLNP below, adapts how much to trust source data and provably avoids negative transfer. Both papers motivate our concern and supply our main competitor. In both, however, the shift is environmental, for example regional climate differences. No adversary chooses the source points to do maximal harm while looking similar.

The mimicry literature supplies the attack logic. A Game-Theoretical Approach for Finding Optimal Strategies in an Intruder Classification Game proves that a strategic intruder's best evasion is to mirror a truncated version of the benign distribution. Our attack is the same idea moved from test-time behavior to calibration data: inject a truncated slice of the benign upper tail. Faking Fairness via Stealthily Biased Sampling shows that subsampling chosen by minimum-cost flow, staying close in Wasserstein distance to the true data, drives an auditor's rejection rate down to its 0.05 significance level on COMPAS and Adult. We reuse that construction to make our injected sets as stealthy as possible against a global audit.

Two results constrain the defense design. The Sample Complexity of Robust Covariance Testing shows that detecting spread-out contamination in $d$ dimensions needs on the order of $d^2$ samples. Auditing in raw feature space is therefore hopeless, and we audit only the one-dimensional score. Towards the Robustness of Differentially Private Federated Learning documents an "outlier paradox": when honest data is noisy, outlier filters such as Krum discard honest points and keep camouflaged ones. This warns that filtering the pool will tend to delete exactly the legitimate tail points the threshold depends on.

The closest game-theoretic work, Game-Theoretic Neyman-Pearson Detection to Combat Strategic Evasion, models an attacker who distorts test-time features against an NP intrusion detector. It assumes known distributions with fully overlapping support. Our attacker never touches test inputs; it poisons the data the threshold is estimated from. Adversarial Cost-Sensitive Classification plays a moment-matching game over label distributions but assumes the training features are authentic, which is exactly the assumption we drop.

Core Claim

Let $s(x)$ be a fixed detector score. The defender calibrates on a pool of $n$ benign scores, of which a fraction $\epsilon$ were chosen by the adversary. The adversary is constrained to values inside the benign support that pass a global two-sample audit.

If all $\epsilon n$ injected points sit just above the clean $(1-\alpha)$ quantile, the empirical $(1-\alpha)$ quantile of the pool lands near the clean $(1-(\alpha-\epsilon))$ quantile. The realized false-alarm rate falls from $\alpha$ to roughly $\alpha-\epsilon$, and the miss rate rises by the probability mass that the dangerous class places in that score band. With $\alpha=0.05$ and $\epsilon=0.03$, the detector is effectively running at a 2% false-alarm rate.

A Kolmogorov-Smirnov audit sees a distribution-function discrepancy of about $\epsilon$. That discrepancy becomes detectable only when $n \gtrsim (c/\epsilon)^2$, with $c\approx 1.36$ at level 0.05, because the test's critical value shrinks as $1/\sqrt{n}$. For $\epsilon=0.03$ this means roughly 2,000 points, more than typical NP calibration sets.

A test restricted to the tail does much better. It counts how many trusted benign points exceed the pool threshold, a binomial count with variance $\alpha(1-\alpha)/m$ for $m$ trusted points instead of the KS test's $1/(4n)$ scale. This gain of roughly a factor of $\alpha$ in variance is what makes a small trusted set useful.

Research Program

The first part characterizes the attack. We will implement tail mimicry against standard order-statistic NP calibration in two forms. The plain form duplicates or resamples real benign tail points. The stealth-optimized form uses minimum-cost flow, in the style of Faking Fairness, to minimize Wasserstein distance to the clean pool subject to a target threshold displacement. We also implement the opposite, alarm-fatigue attack, which injects plausible low-score benign points to pull the threshold down and break the Type-I cap. The asymmetric cost model predicts that a rational adversary prefers the miss-inducing direction, because it violates no monitored guarantee. We will measure how much detection each attack destroys per unit of stealth.

The second part builds the defense, tail-audited NP calibration. Under $\epsilon$-contamination, the clean pool's exceedance rate at a candidate threshold $t$ is bracketed by the observed pool exceedance $\hat F_{pool}(t)$:

$$\frac{\hat F_{pool}(t)-\epsilon}{1-\epsilon} \;\le\; F_{clean}(t) \;\le\; \frac{\hat F_{pool}(t)}{1-\epsilon},$$

where $F_{clean}(t)$ is the true benign false-alarm rate at $t$. The rule works in four steps. It intersects this bracket with a one-sided binomial confidence interval from the $m$ trusted points. When $\epsilon$ is unknown, it estimates the tail contamination from the gap between the trusted and pool exceedance counts. It then selects the lowest threshold whose combined upper confidence bound on $F_{clean}$ stays at or below $\alpha$. Choosing the lowest admissible threshold means erring toward alarms, the cheap error.

We will prove that the rule's Type-I violation probability is at most a chosen $\delta$. We will also bound its excess miss rate relative to an oracle that knows the clean pool. The bound should shrink like $\sqrt{\alpha/m}$ in the tail-audit term, rather than the trusted-only rate. An extension handles poisoning of the dangerous-class training data as well, which affects only the miss rate and so stays a secondary concern.

Baselines

The naive reference point is standard order-statistic NP calibration on the contaminated pool, exactly as a practitioner would run it. It shows the size of the damage. The guarantee-preserving reference is trusted-only calibration, the same order-statistic rule applied to just the $m$ trusted benign points with the same violation probability $\delta$. It is immune to poisoning but must pick a conservative order statistic when $m$ is small. Beating it on miss rate at equal Type-I control is the minimum bar for the pool to be worth using.

The strongest existing method is TLNP (Transfer Neyman-Pearson Algorithm for Outlier Detection), run with the trusted set as target and the poisoned pool as source. This is the right competitor because its adaptive weighting is designed to discount unhelpful source data without knowing the domain similarity. TLNP reports a Type-II error of 0.26 against 0.30 for target-only on ClimSim clusters 26 and 27, and 0.37 on NASA POWER, matching target-only, where pooling exceeds 0.60. We will rerun TLNP with its released tuning heuristics and the same $\alpha$, $\delta$, $m$, $n$ and score model as our method. We will reproduce its clean numbers before adding poison, and give both methods the same small grid for any free constants.

Beating TLNP requires two things. First, no Type-I violation above the nominal $\delta$. Second, a lower clean miss rate, by more than its seed-to-seed spread, at contamination levels of 1–5%. The null-shift NP method of Neyman-Pearson Classification under Both Null and Alternative Distributions Shift is run the same way as a second transfer competitor.

A filter-then-calibrate baseline removes pool points flagged by a score-space outlier detector or by a KS or maximum mean discrepancy screen against the trusted set, then calibrates as usual. Two ablations isolate our key ingredient. One replaces the tail binomial audit with a global KS audit. The other uses a known-$\epsilon$ worst-case rank shift without the trusted audit. Together they show whether the gain comes from looking only at the tail.

Why not just…?

Why not just calibrate on the trusted set and ignore the pool? This is the most serious alternative, because it is trivially safe. Its cost is variance. An order-statistic threshold with violation probability $\delta$ on $m$ points must sit well above the true quantile when $m$ is in the tens to low hundreds, and that inflates misses. TLNP's own results show target-only losing to well-used source data on ClimSim (0.30 against 0.26 Type-II). Our sweep over $m$ will report the crossover point at which trusted-only catches up. If the crossover arrives at $m$ values a practitioner could easily collect, we will say the defense is unnecessary in that regime.

Why not just filter the pool for anomalies first? Tail mimicry injects genuine benign values, so no point-level outlier test can flag them individually. The DP-FL outlier paradox (Towards the Robustness of Differentially Private Federated Learning) reports Krum reaching a near-100% attack success rate on FEMNIST because filters discard honest high-variance points. We expect the same here: filters trim the legitimate tail and bias the threshold. The filter-then-calibrate baseline tests this directly.

Why not just use TLNP, which already guards against bad source data? TLNP's guarantee covers source data that is unhelpful. It does not cover source data that is helpful-looking yet chosen to move one quantile, and its constants are known to be conservative. It may nonetheless be robust in practice. If poisoned-source TLNP matches our rule, the contribution reduces to the attack analysis and a recommendation to always calibrate through TLNP with a trusted target. We will report that outcome plainly.

Why not just raise the cost weight on misses? Reweighting the loss changes the score model, not the calibration step that is poisoned, and the threshold is still read from the contaminated pool. Faking Fairness shows that heuristic adjustments carry no guarantee once data is strategically chosen. We include cost-weighted training followed by naive calibration as an extra arm to confirm this.

Proposed Experiments

Settings. We test four settings. The first is 10-dimensional synthetic Gaussians as in the null/alternative-shift NP paper, where clean quantiles are known exactly. The second is ClimSim clusters 26/27 and 38/37/39, and the third is NASA POWER heavy-rain detection; both reuse TLNP's protocols, so clean baselines can be checked against reported numbers. The fourth is the KDD Cup 1999 login features from the game-theoretic NP paper, a setting where benign logs are naturally untrusted.

Design. We vary $\epsilon \in \{0.01, 0.02, 0.03, 0.05\}$, $n$ from 200 to 5,000, $m$ from 30 to 300, and $\alpha \in \{0.05, 0.1\}$, using a single fixed score model (logistic regression and gradient boosting) per dataset.

Metrics. We report the clean Type-I violation frequency over 500 repetitions, which must stay at or below $\delta$; the clean Type-II error; the global audit's rejection rate on attacked pools, which should sit near 0.05 for a successful mimicry attack; and the expected cost under miss-to-alarm cost ratios of 10 and 100.

Decisive checks. There are three. First, the naive rule's miss rate should rise by a margin that tracks the dangerous-class mass between the clean $(1-\alpha)$ and $(1-(\alpha-\epsilon))$ quantiles, while KS rejection stays near chance for $n \lesssim (1.36/\epsilon)^2$. Confirming this scaling validates the threat model. Second, tail-audited calibration should beat trusted-only and TLNP on clean miss rate at matched Type-I control for small $m$, and the global-audit ablation should lose most of that gain. Third, under the alarm-fatigue attack, the naive rule should violate $\alpha$ while ours should not. If the first check fails on real data, meaning the score band between quantiles carries negligible dangerous-class mass, the attack is cosmetic and the project stops there.