Exact Bounds on Discovery Significance Across All Reasonable Event Selections

When a rare-event search sees only a handful of candidate events, the reported significance depends heavily on which events the analyst chose to keep. An analyst might tighten one cut, loosen another, or shift a classifier threshold, and a 3σ excess can become 2σ or 4σ. This proposal computes, exactly, the largest and smallest discovery significance that any selection in a stated, physically reasonable family could have produced on the same data. If that interval straddles a threshold such as 3σ or 5σ, the claimed evidence depends on the analyst's choice rather than on the data. The idea adapts Matching Bounds, which uses mixed-integer programming to bound causal estimates over all acceptable matchings, to the counting statistics of particle and astroparticle searches.

The technical point is that the obstacle seems to be the nonlinear Poisson significance, and the regime itself removes it. Significance is a convex function of two numbers: the observed count and the expected background. Maximizing a convex function over discrete selections is hard. In the low-statistics regime the observed count is a small integer, though. Once that integer is fixed, maximizing significance reduces to minimizing a linear background sum, which a mixed-integer linear program (an optimization over yes/no decisions with linear objective and constraints, solvable to certified global optimality) handles directly. The scarcity that limits sensitivity is the same property that makes exact bounding tractable.

Core Claim

Consider a family of event selections. Each is a conjunction of thresholds on k reconstructed features, which may include a fixed neural-network score, and each must meet stated constraints: background efficiency at most ε and signal efficiency at least η, both measured on held-out simulation. For any such family, the exact range $[Z_{\min}, Z_{\max}]$ of the Poisson likelihood-ratio significance on the observed data can be computed by solving roughly $N_{\text{obs}}$ linear mixed-integer programs, where $N_{\text{obs}}$ is the number of observed events that could possibly be selected. This avoids piecewise approximation of logarithms. For a selection that keeps $n$ observed events and has expected background $b$ (a weighted sum of simulated background events), the counting-experiment discovery statistic is

$$q(n,b) = 2\left[n \ln\frac{n}{b} - (n-b)\right] \quad (n > b), \qquad Z = \sqrt{q}.$$

For fixed $n$, $q$ decreases as $b$ grows. Therefore $Z_{\max} = \max_n \sqrt{q(n, b^{\min}_n)}$, where $b^{\min}_n$ is the smallest achievable background among feasible selections that keep exactly $n$ observed events. The quantity $b^{\min}_n$ is a linear objective subject to linear constraints. $Z_{\min}$ follows symmetrically from $b^{\max}_n$. This is the "sequence of linear subproblems" strategy that Miyauchi et al. used for fractional modularity density. Here it is applied to a statistical discovery criterion and made exact by the integrality of $n$.

Relation to Prior Work

Matching Bounds supplies the template and motivates the question. It bounds a difference-in-means treatment effect over all matched samples meeting a covariate-balance tolerance, and on ten replicated political-science studies it found that in 90% of tests some matching method was significant while another was not. Its estimand is linear, it does not handle Poisson likelihoods or significance thresholds, and the authors flag scalability as the bottleneck. We change the estimand to a nonlinear counting significance. We also replace "all matchings" with a structured selection class defined on features, not on individual events. This matters: bounding over arbitrary event subsets would only measure how much one could cherry-pick, which is uninformative.

Serrano and Vidal supply the contrasting method. They solve a nonlinear likelihood exactly by piecewise outer approximation of the logarithms, big-M constraints (linear constraints with a large constant that switch terms on or off according to a binary variable), and cutting planes. That approach stopped at 20-node graphs. Our enumeration over $n$ is meant to avoid the approximation and its growth in problem size, and we test this directly against their style of formulation.

The physics literature motivates the regime. Telescope Array's hotspot analysis and the ψ(2S) branching-fraction study (Branching fractions for ψ ( 2 S ) → γ η ′ and γη) show borderline 2–3σ results that hinge on small counts and analysis choices.

Baldi et al. and EXO-200 show that learned discriminants raise sensitivity. We do not compete with them. Their scores enter our selection class as one more feature, and we ask how fragile the final cut on top of them is.

Research Program

The work has four parts.

First, we build the formulation. Binary variables mark which candidate threshold is active on each feature. Each observed and simulated event gets an inclusion indicator, linked to the thresholds by linear constraints so that an event is included exactly when it lies inside every bound. This is the standard encoding used in optimal decision-tree programs. Background and signal efficiency are linear sums over weighted simulated events, and the observed count is fixed per subproblem.

Simulation samples run to millions of events, so we cannot carry one indicator per simulated event. Instead we restrict candidate thresholds to the observed-event coordinates plus simulation quantiles, and we compress the simulation into the cells this grid induces. The compression is exact for the chosen grid. Grid resolution is reported as an explicit approximation, and its effect is measured separately from the exactness of the optimization. Subproblems for neighboring $n$ warm-start one another.

Second, we use the exact maximum for calibration. When thresholds are tuned on the data, the correct look-elsewhere correction (the adjustment for having tried many selections) needs the null distribution of the true maximum over the class. A heuristic optimizer that misses the maximum on background-only toy experiments produces a correction that is too lenient whenever the real analysis searched harder than the toys did.

Third, we extend the selection class beyond single boxes to unions of two boxes and to thresholds on monotone score combinations. This tests how far exactness survives as the class grows.

Fourth, we release a solver-backed tool that reports, alongside any significance claim, its interval and the extreme selections that achieve each end.

Baselines

The naive reference point is the single fixed selection that a standard analysis would report. Following Baldi et al., this is a cut on a deep network's score chosen to maximize expected significance on simulation. That paper reports 4.9σ for the deep network versus 2.5σ for a shallow network using low-level HIGGS features. We reproduce this point estimate on our pseudo-experiments. It is the number the interval brackets. The proposal does not try to beat it, only to show how much it could move.

The strongest practical competitor for estimating the range is heavy stochastic search: random sampling plus Bayesian optimization over the same threshold grid and constraints. Each method gets the same wall-clock budget as the solver on identical pseudo-experiments and simulation splits, and each method's tuning is chosen on separate validation toys. For two features, exhaustive grid enumeration is also feasible and serves as ground truth. We beat stochastic search if, at k of 4 or more, it reaches less than the certified maximum (or more than the certified minimum) in a meaningful fraction of pseudo-experiments, especially ones near a significance threshold. No reported result was found on Lacuna for selection-range estimation on HIGGS, SUSY, or CMS H(bb̄). The only published fragility figures are the 90% and 62% significance and sign flips in Matching Bounds, which are on a different problem.

The ablation that isolates the key ingredient keeps everything else fixed but replaces the enumeration over $n$ with a single program that approximates the logarithm by piecewise outer bounds and big-M constraints, as in Serrano and Vidal. We compare runtime, optimality gap at timeout, and the largest $N_{\text{obs}}$ and k each version solves to certified optimality. A second ablation removes the efficiency constraints. It should produce intervals so wide they are meaningless, which confirms that the constraints, not the solver, carry the scientific content.

Why not just…?

Why not just scan cuts on a grid? For one or two features, exhaustive scanning works, and we use it to verify our solver. The cost grows as the grid size raised to twice the number of features, so with four to six features it becomes infeasible. That is where the comparison against stochastic search decides the question. If tuned random search consistently recovers the exact endpoints within a few hundredths of a sigma, the honest conclusion is that the solver adds only a certificate, and we will report it that way.

Why not just run a multiverse analysis over a dozen analyst-chosen variants? A handful of variants is a sample, and the extreme selections are rarely the ones a person writes down. Matching Bounds found exactly this pattern with matching heuristics on real studies: published variants disagreed, and the full bounds were wider still. We measure what fraction of the exact interval width a twelve-variant multiverse recovers.

Why not just apply a look-elsewhere trials factor? A trials factor corrects the false-alarm rate of one search procedure. It does not say whether a different reasonable procedure would have reached the opposite conclusion. It is also only valid if the toy experiments use the same maximization as the real analysis, and we argue that maximization should be exact. Experiment two below tests whether approximate maximization makes the correction too lenient.

Why not just train a better classifier and cut once? Better discrimination shifts the whole interval upward, as the gains reported by EXO-200 and Baldi et al. suggest. It does not remove the fragility of the final threshold at small counts. Because we include the network score as a feature, we can show directly how much interval width remains after the best available discriminant is used.

Proposed Experiments

We build pseudo-experiments from the HIGGS and SUSY benchmarks and the CMS H(bb̄) open dataset. Each benchmark is split into a simulation half, used for background and signal efficiencies and for training the network score, and an "observed" half. From the observed half we draw small datasets with expected signal of 2–20 events over background of 1–50 events, giving thousands of toy experiments per configuration. The features are the network score plus three to five high-level kinematic variables. The efficiency constraints ε and η are set around the values achieved by the optimized reference cut.

The first decisive check is correctness and exactness. With two features, solver endpoints must match exhaustive enumeration exactly. At four to six features, we report how often stochastic search fails to reach the certified endpoints.

The second check is scientific. It measures the fraction of toys where the reference cut reports at least 3σ but $Z_{\min}$ falls below 3σ, as a function of expected counts and of how much ε and η are relaxed. This mirrors the 5–10% balance relaxation that flipped the trade-openness finding in Matching Bounds.

The third check is calibration. On background-only toys, we compare the null distribution of the exact $Z_{\max}$ with the heuristic maximum and report the resulting difference in post-trial p-values.

The fourth check is scaling. We report runtime against $N_{\text{obs}}$, k, and grid resolution, and the ablation against piecewise outer approximation. The proposal fails on scaling if exactness collapses beyond about 50 observed events or 4 features. That outcome would confine the method to the very-low-count regime it targets.