Many scientific pipelines score each measurement for reliability and discard the ones that score poorly. Examples include satellite retrievals with too much cloud, altimetry waveforms that tracked badly, and gene imputations a model is unsure of. In large datasets this costs little. In searches for rare events the cost is steep, because the uncertainty on a count of $N$ events shrinks only as $1/\sqrt{N}$. Every discarded event is paid for directly in statistical significance. This note argues that "keep or discard" is the wrong question. If the background model is correct, a likelihood that conditions on each event's reliability score and weights events accordingly never loses to discarding them. Discarding pays off only when the model for low-reliability events is wrong in ways nobody parameterized.
The proposal makes that precise and tests it. Treat a reliability gate as the limiting case of a nuisance parameter, an unknown quantity such as a background normalization that the fit is allowed to adjust, whose allowed range is infinite. Estimate the real size of that uncertainty separately for each reliability stratum from signal-poor control data. Then check, on collider benchmarks shrunk down to rare-event counts, whether this beats hand-tuned gates and significance-optimized classifiers. It should beat them both on expected discovery significance and on how often it claims a discovery that is not there.
Core Claim
Current work that ties classifiers to discovery significance treats selection as a hard cut. Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge turns the Approximate Median Significance (AMS) into a sequence of reweighted classification problems. AMS is the particle-physics measure of how surprising a count excess is over a known background. The method still ends in a single accept region. Consistent optimization of AMS by logistic loss minimization shows that training with ordinary logistic loss and then tuning a threshold is consistent for AMS. Its bound scales with $s/b$ at the optimum, so it becomes vacuous exactly when the background is pushed toward zero.
On the reliability side, Imputing spatial transcriptomics through gene network constructed from protein language model keeps only the top 50% most reliable imputed genes, raising average cosine similarity from 0.67 to 0.74. Adaptive Feature Imputation with Latent Graph for Deep Incomplete Multi-View Clustering accepts an imputation only if clustering confidence does not drop. Both scores measure local fidelity, and neither is ever connected to a downstream test. These papers motivate the question; they supply no method here.
The claim has two parts. The first is a statement about optimality that should hold whenever the background model is correct. Write $x$ for an event's features and $r$ for its reliability score. Write $s(x,r)$ and $b(x,r)$ for the expected signal and background densities, $S$ and $B$ for their totals, and $\mu$ for the signal strength being tested. Then the extended Poisson likelihood
$$\ln L(\mu) = -(\mu S + B) + \sum_i \ln\big(\mu\, s(x_i,r_i) + b(x_i,r_i)\big)$$
uses every event, and dropping events can only lose information. Here "correct" means the densities are well specified.
The second part is operational. Split events into reliability strata $k$, and give each stratum's background a normalization nuisance with fractional width $\epsilon_k$. Discarding stratum $k$ is then the same as setting $\epsilon_k \to \infty$. The useful question is therefore how large $\epsilon_k$ really is, and that can be measured from control regions instead of guessed. The prediction follows: data-driven $\epsilon_k$ will keep marginal strata that heuristic gates throw away, and will shrink toward the gate only where control data show genuine mismodeling.
Research Program
The first component builds reliability-conditioned density-ratio estimators. These are classifiers whose output approximates $s(x,r)/b(x,r)$, with the reliability score as an input rather than a filter. Two reliability scores will be studied. One is ensemble disagreement among independently trained classifiers. The other, where features are missing or imputed, is the imputer's own uncertainty, following the post-hoc reliability scorers above.
The second component estimates $\epsilon_k$ per stratum. The fit is run on signal-depleted control regions, and the residual disagreement between simulated and observed background yields is measured in each stratum. That spread, with its own statistical error from the finite control sample, becomes the constraint width in the profile likelihood.
The third component is a small-count study that addresses the validation problem raised by Mackey et al. and Kotłowski. Every tuning choice, including the choice of strata, is fixed on simulation. Only the $\epsilon_k$ are touched by control data, and never by the signal region. This avoids the "hallucinated significance" that Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge observed when weights were tuned on the same data used for evaluation.
The regime of interest is the one illustrated by Energy and flux measurements of ultra-high energy cosmic rays observed during the first ANITA flight, whose flux rests on 16 events, and by the Telescope Array hotspot in INDICATIONS OF INTERMEDIATE-SCALE ANISOTROPY OF COSMIC RAYS WITH ENERGY GREATER THAN 57 EeV IN THE NORTHERN SKY MEASURED WITH THE SURFACE DETECTOR OF THE TELESCOPE ARRAY EXPERIMENT, which stays below $5\sigma$. We cannot rerun those experiments. We instead emulate their counts by subsampling large simulated collider datasets down to expected signal yields of a few to a few tens of events.
Search for Neutrinoless Double-β Decay with the Complete EXO-200 Dataset supplies the motivating precedent. It replaced heuristic single-site/multi-site cuts with a continuous network score and gained about 25% in half-life sensitivity. However, it still applied selection cuts up front, and its limit was simulation–detector mismatch. That is exactly the misspecification our $\epsilon_k$ are meant to quantify.
Datasets and Stress Tests
The main testbed is the FAIR Universe HiggsML Uncertainty Dataset and Competition release: 28 features, 280 million training events, and six parameterized nuisance parameters. It asks for confidence intervals on signal strength, which is the quantity we report. That paper explicitly leaves unmodeled corruption open. We create such corruption in a controlled way by applying shifts outside the six nuisances only to events in low-reliability strata, at strengths the method does not know. This yields cases where gating should win, cases where keeping everything should win, and cases in between.
The 2014 Kaggle HiggsML data, used in both AMS papers, is the second testbed. It allows direct comparison with the cascade on the AMS metric at full statistics. It is also the setting where the file reports numbers.
Baselines
The naive reference point trains a gradient-boosted classifier with logistic loss on all events, applies no reliability information, and tunes a single threshold for AMS on simulated validation data. This is the two-stage procedure that Consistent optimization of AMS by logistic loss minimization justifies. It is the strongest simple method in large-count regimes. It is run on identical training data with the same tree-model family and hyperparameter budget as every other arm. Beating it means higher median expected significance at matched false-discovery rate across the subsampled yields.
The strongest existing significance-oriented method is the weighted classification cascade of Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge. On the Kaggle private leaderboard it reports an AMS of 3.72594, against 3.67 for its unweighted ensemble baseline. We reimplement it with the same base learner and number of cascade rounds, and tune it on held-out simulation as its authors require. On Kaggle, matching 3.72594 within run-to-run noise at full statistics is the sanity bar; our claim is not about beating it there. The claim concerns the small-count and corrupted regimes, where beating it means higher expected significance together with a nominal-level false-discovery rate. For FAIR Universe there is no reported result found on Lacuna for any baseline's coverage or interval width, so all numbers there come from our own reruns.
The reliability-gated baseline represents Direction 1 practice. It discards events below a reliability threshold, tunes that threshold for expected significance on simulation, and then runs the naive pipeline. The gating strength is swept across the same range our strata cover.
The ablation that isolates the key ingredient is the full stratified profile likelihood with the $\epsilon_k$ replaced by a single hand-set width shared across strata. This follows standard practice: the analyst assigns one generic systematic. If data-driven, stratum-specific widths do not outperform this ablation under the injected corruptions, the proposal reduces to existing profile-likelihood practice and the claim fails. A second ablation removes $r$ from the classifier inputs while keeping the stratification, to test whether conditioning or stratification carries the gain.
Why not just…?
Why not just cut on the classifier output and stop? In large samples, Kotłowski's consistency result says this is close to optimal. Our method should roughly tie with it there, and we will report that tie. The two diverge in the $b \to 0$ regime where that bound becomes vacuous, and when low-reliability events are mismodeled. A single cut cannot distinguish a confident event from a well-reconstructed one. The small-count and corruption sweeps test exactly this.
Why not just tune a hard reliability gate on validation data? In real rare-event searches there is almost no validation data. With 16 events, any split leaves both halves dominated by Poisson noise, and tuning on the evaluation data inflates significance. We therefore tune the gate on simulation, which is fair but assumes simulation is right. Where simulation is wrong, a simulation-tuned gate cannot detect it, whereas control-region $\epsilon_k$ can. The corruption experiments settle this directly.
Why not just put reliability bins into a standard profile likelihood with generic per-bin nuisances? This is a real threat. It is close to what careful physics analyses already do. Our method is that analysis plus one ingredient: widths estimated per stratum from control data instead of assigned by hand. The shared-width ablation is designed so that if hand-set widths match data-driven ones, we report that the contribution is negligible. The idea earns its place only if data-driven widths correctly recover the injected, stratum-specific mismodeling while keeping clean strata.
Why not just use the cascade, which already optimizes significance? The cascade optimizes a counting metric for one accept region under fixed simulation weights. It has no mechanism for per-event reliability or for simulation error. We run it in every regime and expect it to be competitive at full statistics.
Proposed Experiments
The decisive experiment is a grid over expected signal yield, from full FAIR Universe statistics down to about five expected signal events, crossed with injected low-reliability corruption strength from zero to severe. Every method runs on the same simulation for training and tuning and the same pseudo-experiments for evaluation. Each cell reports three quantities: median expected significance, empirical coverage of the 68% interval on signal strength, and the rate of $3\sigma$ excesses in pseudo-experiments with no injected signal.
The note's central prediction is falsifiable and has two sides. At zero corruption, the stratified likelihood keeps essentially all events and beats every gate at small yields, and the gain should grow as yields shrink because each discarded event costs more there. At strong corruption, the estimated $\epsilon_k$ for the corrupted strata should grow until those strata are effectively discarded, so the method behaves like the best tuned gate without being told which strata are bad. Throughout, it should hold coverage where the naive pipeline and cascade lose it.
A method that fails on either side, by gating too little under corruption or too much without it, would show that measured nuisance widths cannot replace judgment about when to throw data away. The Kaggle comparison against the cascade's reported 3.72594 serves only as a check that the reimplementations are sound.
References
- Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge
- Consistent optimization of AMS by logistic loss minimization
- Imputing spatial transcriptomics through gene network constructed from protein language model
- Adaptive Feature Imputation with Latent Graph for Deep Incomplete Multi-View Clustering
- Energy and flux measurements of ultra-high energy cosmic rays observed during the first ANITA flight
- INDICATIONS OF INTERMEDIATE-SCALE ANISOTROPY OF COSMIC RAYS WITH ENERGY GREATER THAN 57 EeV IN THE NORTHERN SKY MEASURED WITH THE SURFACE DETECTOR OF THE TELESCOPE ARRAY EXPERIMENT
- Search for Neutrinoless Double-<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>β</mml:mi></mml:mrow></mml:math>Decay with the Complete EXO-200 Dataset
- FAIR Universe HiggsML Uncertainty Dataset and Competition