Certified Discovery Significance When Background Counts Run Out

Particle physics searches for rare signals, such as the Higgs boson or neutrinoless double-beta decay, train a classifier to separate signal from background. They then keep only events above a score cutoff and judge the result by a discovery significance: roughly, how many standard deviations the surviving signal sits above the surviving background. The tighter the cutoff, the purer the selected region. But a tight cutoff also leaves very few simulated background events in that region. The background estimate there is then a count of perhaps 0 to 10 events, and its relative error grows like $1/\sqrt{k}$ for $k$ surviving events. Standard practice picks the cutoff that maximizes the estimated significance on a validation set. In the sparse regime this rewards cutoffs where the background count happened to fluctuate low. The method then reports significance the data do not support. Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge explicitly warns that its cascade "hallucinates significance" unless weights are validated on held-out data.

We propose choosing the cutoff by maximizing a high-probability lower bound on the true significance. The bound replaces the background estimate with an exact Poisson upper confidence limit computed from the number of calibration background events that survive the cut. The confidence level is corrected for the fact that many cutoffs were tried. The asymmetry is the point. Overstating significance means a false discovery claim, which is far costlier than a modestly conservative cut. Unlike the empirical-constraint tightening used in Neyman-Pearson classification, the bound never becomes infeasible: zero surviving background events still give a finite upper limit. The sample-size floor therefore turns into something useful, a ceiling on how much significance a given simulation budget can certify.

Core Claim

Selecting the operating point by a multiplicity-corrected Poisson lower bound on significance, instead of the plug-in estimate, does three things. It removes the optimism gap between reported and true significance in the low-background regime. It recovers most of the plug-in method's true significance when calibration data are plentiful. It beats the standard fixed-regularizer fix when calibration background is scarce. The key ingredient is that the conservatism adapts to the per-cutoff background count $k(t)$, rather than coming from a constant.

Relation to Prior Work

Two 2014 papers supply the setting and motivate the gap. Mackey et al. turn maximization of the Approximate Median Significance (AMS, a Poisson profile-likelihood significance of selected signal $s$ against background $b$) into a sequence of reweighted classification problems. They raised the HiggsML private leaderboard AMS from 3.67 to 3.72594. Their analysis assumes large simulated samples and offers no finite-sample guarantee on the reported significance.

Consistent optimization of AMS by logistic loss minimization proves that the simple recipe is consistent: train a logistic model, then tune the cutoff on validation data. Its regret bound carries a prefactor $s/b$ that diverges as $b \to 0$, and threshold selection adds $O(1/\sqrt{m})$ variance for $m$ validation points. Our proposal does not change the classifier. It targets exactly the threshold-selection step where that bound becomes vacuous.

Three Neyman-Pearson papers constrain the design. Neyman-Pearson classification under a strict constraint tightens the empirical error constraint by $\tau/\sqrt{n}$ to certify the population constraint. At small $n$ that tightened constraint becomes empty. Neyman-Pearson classification: parametrics and sample size requirement shows the distribution-free floor $n_0 \ge \ln(1/\delta)/\alpha$. Its nonparametric method returns "Unable to Process" on a 118-sample cancer dataset, and the paper escapes the floor only through a Gaussian assumption. Neyman-Pearson Multi-class Classification via Cost-sensitive Learning inherits similar small-count fragility.

These results tell us not to promise distribution-free certification of tiny error rates, which is impossible. We certify a quantity that degrades gracefully instead. Significance is monotone in $b$, so an upper limit on $b$ always yields a valid, if smaller, significance.

The FAIR Universe challenge (FAIR Universe HiggsML Uncertainty Dataset and Competition) supplies the evaluation data. It is a different task, building confidence intervals on signal strength under nuisance parameters. Its winners report Quantile Scores of 0.878 (HEPHY) and 0.823 (IBRAHIME), which do not measure the quantity we optimize.

Research Program

Fix a trained scoring function and a family $T$ of candidate cutoffs. For each cutoff $t$, a calibration set kept separate from training gives a lower confidence limit $s_{\mathrm{lo}}(t)$ on selected signal, and $k(t)$ surviving background events, each carrying a simulation weight. We define the certified significance as

$$ Z_{\mathrm{cert}}(t) = \mathrm{AMS}\big(s_{\mathrm{lo}}(t),\; b_{\mathrm{up}}(t)\big), \qquad b_{\mathrm{up}}(t) = \bar w(t)\, U_{\delta/|T|}\big(k(t)\big), $$

Here $U_{\gamma}(k)$ is the exact Poisson upper limit at confidence $1-\gamma$ for observed count $k$; for example, $U \approx 3$ at $k=0$ and 95%. The factor $\bar w(t)$ is the mean background weight among surviving events. The confidence level is divided by $|T|$, a union bound over cutoffs that plays the role of a look-elsewhere correction. We select $\hat t = \arg\max_t Z_{\mathrm{cert}}(t)$.

AMS increases in $s$ and decreases in $b$. So on the event that all limits hold, which has probability at least $1-\delta$ by the union bound, the true significance at $\hat t$ is at least $Z_{\mathrm{cert}}(\hat t)$. This holds simultaneously over all $t$, so it also holds at the data-chosen cutoff.

We will develop three refinements. The first handles unequal event weights through an effective-count Poisson approximation, and we will measure whether the bound's coverage survives it. The second replaces the crude union bound with a bound that exploits nesting: the cutoffs are ordered, so $k(t)$ is monotone, and a correction that uses this should cost less than one over $|T|$. The third is a simulation-budget curve. It predicts the maximum certifiable significance as a function of calibration background size, which tells an analysis how much more background simulation to generate.

Finally, we will plug $Z_{\mathrm{cert}}$ into the outer loop of the Mackey et al. cascade, in place of plug-in AMS, as the criterion for accepting a weight update. This tests whether the cascade's overfitting is mainly a selection artifact.

The expected behavior is as follows. The suboptimality of $\hat t$ relative to the true-optimal cutoff should scale with the Poisson relative error $1/\sqrt{k(t^\ast)}$ at the optimal cutoff, inflated by a $\sqrt{\log(|T|/\delta)}$ multiplicity term, which is the usual form of a union-bounded concentration inequality. It should not blow up with $s/b$ as in Kotlowski's bound. Proving a clean version of this is a theory goal. Measuring it is an empirical one.

Baselines

The naive reference is unweighted XGBoost with a fixed selection fraction chosen once on the full data and never re-tuned. It is the "do nothing" point and indicates how much threshold choice matters at all. Mackey et al. report a related unweighted XGBoost ensemble at private AMS 3.67 on HiggsML, which anchors our implementation.

The strongest practical competitor is post-hoc threshold tuning with plug-in AMS and the challenge's fixed background regularizer $b_r = 10$, justified by Kotlowski's consistency result. It is what essentially every HiggsML entry did, and it is asymptotically optimal. The strongest training-side method is the Mackey et al. weighted cascade, with reported private AMS 3.72594. We run it with its held-out weight validation.

For fairness, all methods share the same trained scoring function, or the same cascade for the cascade variant. They also share identical calibration subsamples, the same candidate cutoff grid, and the same tuning budget. In particular, $b_r$ for the regularized baseline is itself tuned on the calibration set, with no access to the evaluation set.

Two ablations isolate the key ingredient. The first uses our bound without the multiplicity correction. The second replaces the exact Poisson limit with a Gaussian $k + z\sqrt{k}$ limit, which collapses to zero at $k=0$.

Beating the baselines means two things. Our method must achieve higher mean true AMS than plug-in tuning in the scarce-calibration regime. It must also keep the reported-minus-true optimism gap non-positive in at least $1-\delta$ of runs, a property plug-in tuning should violate. In the plentiful regime we must stay within a small margin of plug-in true AMS.

Why not just…?

Why not just tune the threshold on a held-out set, as Kotlowski justifies? This is the main threat when calibration data are large. That is why we report the plentiful regime and accept a small loss there. Its guarantee is asymptotic, though. The $s/b$ prefactor and the $O(1/\sqrt{m})$ selection variance both worsen exactly where background is scarce, and maximizing a noisy estimate over many cutoffs systematically picks favorable fluctuations. The subsampling experiment below measures this optimism directly against near-exact true significance.

Why not just keep the fixed $b_r$ regularizer, or tune it? A constant $b_r$ adds the same pessimism regardless of how many background events actually survive. It over-penalizes loose cuts with many events and may under-penalize tight cuts with none. If a tuned $b_r$ matches our method across all calibration sizes, our contribution reduces to a principled way to set it, and we will say so. We expect it not to match, because the best $b_r$ should shift with calibration size. We test this by tuning $b_r$ at each size and checking whether one value works across sizes.

Why not just use a parametric threshold, as pNP-sLDA does below the sample floor? That route assumes Gaussian class-conditional scores. Classifier outputs on collision kinematics are visibly non-Gaussian in the tails, and those tails are exactly the selected region. We include a Gaussian-tail fit to background scores as an additional baseline and measure its coverage failures.

Why not just simulate more background? Often that is the right answer, and the simulation-budget curve is designed to say when. Simulation is expensive at extreme rejection, though, and some searches, such as EXO-200's half-life limits near $10^{25}$ years in Search for Neutrinoless Double-β Decay with the Complete EXO-200 Dataset, have background models constrained by limited calibration data. Knowing how much significance the current budget can honestly certify is itself the deliverable.

Proposed Experiments

On the 2014 HiggsML dataset (30 features with physical event weights), we train scoring functions on one split. We reserve a large split as near-ground-truth for true $s$ and $b$ at every cutoff. We then repeatedly draw calibration subsamples whose background size ranges over orders of magnitude, down to tens of events above the relevant cutoffs. For each method and size we record three quantities: the true AMS at the selected cutoff, the optimism gap, and the empirical coverage of the certified lower bound over many draws.

The FAIR Universe dataset, with 280 million simulated events, allows far tighter ground truth in the extreme tail. There we repeat the protocol at much stricter cutoffs where true $b$ is genuinely tiny. As a stress check, we also apply calibration drawn at one jet-energy-scale setting to evaluation at a shifted setting. This tests whether the guarantee degrades gracefully under the systematic shifts it does not model.

The decisive checks are three. The plug-in method must show a positive optimism gap that grows as calibration shrinks; if it does not, the problem we target is not real. Our bound must hold at its nominal rate. Our true AMS must exceed tuned-$b_r$ and plug-in selection in the scarce regime by more than run-to-run variance. If the Gaussian-limit ablation performs as well as the exact Poisson limit, the zero-count regime is not where the action is, and we will narrow the claim accordingly.