Calibrating the Systematic on the Systematic in Cut-and-Count Anomaly Searches

Collider searches for new particles increasingly use machine-learned filters that keep only one background event in ten thousand. To decide whether the few events that survive are surprising, physicists need an estimate of how many background events should survive. They also need an error bar on that estimate, called a residual systematic uncertainty: the leftover mismatch between the learned background model and real data after every correction has been applied. Current methods summarize this mismatch with a single number, measured where data are plentiful and then assumed to hold where data are scarce. This proposal treats the mismatch as a smooth unknown function of two quantities: the collision mass, and how aggressive the filter is. It fits that function with a Gaussian process, a regression model that places a probability distribution over smooth curves and reports how uncertain it is at each point. Crucially, it averages over the Gaussian process's own tuning knobs, or hyperparameters, instead of fixing them at single best-fit values.

The reason this matters is specific. The most aggressive filter settings are where discoveries happen, and they are also where the mismatch has to be extrapolated from looser settings with almost no direct data. In that regime, a model with fixed tuning knobs is known to be badly overconfident. An overconfident systematic error bar inflates discovery significance. The proposal asks a narrow question that can be checked: on the standard collider anomaly benchmark, with a deliberately wrong background simulator, does averaging over the hyperparameters make the false-discovery rate match its nominal value? And can it do so without the sensitivity loss of worst-case bounds?

Core Claim

Accurate and robust methods for direct background estimation in resonant anomaly detection counts events directly inside the signal window after a learned cut. It uses a weakly supervised classifier (CWoLa) or a conditional density model (CATHODE). This "cut-and-count" approach avoids two failure modes. It avoids background sculpting, where the classifier distorts the mass spectrum into fake bumps. It also avoids the collapse of parametric sideband fits at background efficiencies $\epsilon_B = 10^{-3}$ and $10^{-4}$. Mismodeling enters that paper through a single shift $\delta_{sys}$ with uncertainty $\sigma_{sys}$. The shift is estimated either from a Pythia-versus-Herwig comparison or from data in the sidebands, the mass regions on either side of the signal window.

Our claim is that this estimate is itself a regression problem. Its dominant error is the between-model term identified by Prediction Performance After Learning in Gaussian Process Regression. Write $r(m, u)$ for the relative residual between the observed count and the template prediction. Here $m$ is the dijet mass bin and $u = \log_{10}\epsilon_B$ is the cut tightness. We place a Gaussian process prior on $r$ with kernel hyperparameters $\theta$: one lengthscale along mass, one along tightness, and an amplitude. We fit it to residuals measured in sidebands at all tightness values. The residual uncertainty in the signal window at the working point $u^*$ is then

$$\sigma_{sys}^2(u^*) = \mathbb{E}_{\theta \mid D}\big[\mathrm{Var}(r \mid \theta, D)\big] + \mathrm{Var}_{\theta \mid D}\big(\mathbb{E}[r \mid \theta, D]\big),$$

where $D$ denotes the sideband residuals. The second term vanishes under the usual practice of fixing $\theta$ at its marginal-likelihood maximum.

The tightness lengthscale controls how far well-measured residuals at loose cuts are trusted at the tight cut. The data barely constrain that lengthscale, because at tight cuts there are only a handful of sideband events. So the second term should dominate at exactly the working points where Das et al. report discoveries. The discovery significance then becomes $Z = S/\sqrt{B + B^2\sigma_{sys}^2}$, where $S$ and $B$ are the excess and predicted background counts. Our prediction is that this $Z$ is calibrated under the null, meaning background-only pseudo-experiments produce Z-scores distributed as a standard normal. We also predict it keeps most of the sensitivity of the scalar approach.

A practical point removes the usual objection to full Bayesian treatment. The regression has only tens to a few hundred inputs (mass bins times thresholds), not millions of events. Sampling the hyperparameters with a No-U-Turn sampler, a standard Markov chain Monte Carlo method, costs seconds. The cubic scaling that limited A Gaussian process framework for modelling instrumental systematics: application to transmission spectroscopy is irrelevant here.

Relation to Prior Work

Two papers motivate the claim by establishing that fixing hyperparameters undercovers. Wågberg et al. derive the missing between-model variance. Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications shows the effect is severe with sparse data: maximum-likelihood bounds fail 98% of the time on Sarcos with $N=850$, against 8% for their robust bounds. Neither paper touches hypothesis testing for discovery.

Gibson et al. supply the method template. They used a Gaussian process with full hyperparameter marginalization to model instrument systematics in Hubble light curves, and found that maximum-likelihood error bars were too narrow. We move that idea from the time axis of one instrument to the joint mass-by-tightness axis of a learned background template. The goal is no longer a parameter error bar but a false-discovery rate.

Das et al. supply both the pipeline and the strongest baseline. We keep their counting procedure unchanged and replace only how $\sigma_{sys}$ is produced.

Two lines of work constrain the design. Capone et al. take the worst case over a hyperparameter confidence region. Gap 5 of our analysis points out that inflating $\sigma_{sys}$ this way could erase faint signals, so we test the worst-case variant explicitly rather than adopting it. Robust resonant anomaly detection with NPLM aggregates test statistics across many selection thresholds into a "hyper-test." That aggregation implicitly needs systematics that are correlated across thresholds, which our tightness axis provides. Combining the two is a natural extension but not the core test.

Research Program

We first build the residual-measurement step. The background template comes from CATHODE and CWoLa, following Das et al. On the LHC Olympics R&D sample, it is trained on Pythia background and applied to pseudo-data drawn from Herwig. This creates real, phase-space-dependent mismodeling with a known answer, because large held-out Herwig samples give the true residual at every $(m, u)$.

Residuals are recorded in sideband mass bins over a grid of tightness values from $\epsilon_B = 10^{-1}$ to $10^{-4}$. Each residual gets a Poisson-derived noise variance, so the Gaussian process knows that tight-cut points are noisy. We then fit the Gaussian process three ways: with maximum-likelihood hyperparameters, with sampling under weakly informative priors on the lengthscales, and with the confidence-box worst case of Capone et al. In each case we propagate $\sigma_{sys}$ into $Z$.

Two variations carry most of the scientific content. The first is removing the tightness axis, which fits a separate mass-only Gaussian process at each cut. The second is varying the lengthscale prior, to measure how much the conclusions hinge on it.

The template's own training randomness is a separate source of variance. We handle it by retraining the template on a subset of pseudo-experiments. This tells us whether the residual Gaussian process absorbs that randomness or whether it must be added on top.

Baselines

The naive reference point uses no residual systematic: $\sigma_{sys} = 0$, with only Poisson counting error. It shows how badly the Herwig mismatch alone miscalibrates $Z$.

The strongest existing method is the scalar $\delta_{sys}$ of Das et al., in both its simulator-difference and data-driven forms. It is the strongest competitor because it is the only published treatment of residual systematics in cut-and-count anomaly searches. The paper reports discovery significance above 5σ for both CWoLa and CATHODE at $\epsilon_B = 10^{-3}$ and $10^{-4}$ on the R&D data, against 8σ for an idealized detector. It also reports low significance on background-only tests, but the file gives no reported number for the null false-positive rate. Running it faithfully requires two things. Every method uses identical trained templates, identical pseudo-data, and identical sideband definitions. Every method also receives the same residual measurements as input, so the only difference is the map from residuals to $\sigma_{sys}$.

We beat it if one of two things happens. Either the scalar method's false-positive rate at nominal 3σ visibly exceeds its target under the Herwig mismatch while ours does not. Or both are calibrated and ours loses less median significance on injected signal.

The worst-case confidence-box bound of Capone et al. is the strongest coverage-oriented competitor. We run it with the same priors and the same coverage level. We expect it to be calibrated but conservative, so beating it means comparable null calibration at higher median $Z$.

The Gaussian process with maximum-likelihood hyperparameters is the ablation that isolates our key ingredient. Its mean prediction is the same as ours; it differs only in dropping the between-model term. If it is already calibrated, the hyperparameter story fails.

The mass-only Gaussian process is a second ablation. It isolates the benefit of borrowing strength across tightness. For context on the sensitivity scale, the NPLM paper reports median Z of $0.30 \pm 0.04$ for its hyper-test against $0.1 \pm 0.1$ for a boosted decision tree at $S/B = 8.2\cdot10^{-4}$. That is a different statistic, so we cite it for context rather than as a head-to-head target.

Why not just…?

Why not just inflate the scalar $\delta_{sys}$ by a safety factor, or take the larger of the simulator and sideband estimates? This is the most serious threat, and it may simply work. The question is whether any single factor works across working points. Our hypothesis is that the needed inflation grows as cuts tighten, because extrapolation error grows. A fixed factor tuned at $\epsilon_B = 10^{-3}$ would then undercover at $10^{-4}$, and one tuned at $10^{-4}$ would waste sensitivity at $10^{-3}$. We test this directly by fitting the best single factor on one working point and evaluating calibration at the others. If one factor calibrates all working points, we will report that the simple fix suffices.

Why not just fit a low-order polynomial to the sideband residuals and propagate its parameter covariance? This captures parameter uncertainty within a fixed functional form, but not uncertainty about the form itself. Gibson et al. found that imposing rigid forms on instrument systematics produced overconfident and conflicting conclusions. Das et al. also show that fits starve for events at tight cuts. We include polynomials of degree one to three in mass and tightness as an extra comparison. A polynomial matching our null calibration would reduce the claim to the tightness axis alone.

Why not just ensemble templates over seeds and classifier settings? That approach measures the template's training variance, which the NPLM work shows is large. It does not measure the bias that comes from training on the wrong simulator, because every ensemble member shares that bias. Our template-retraining subset quantifies this split directly. We will report how much of the null miscalibration each source explains.

Why not just sample hyperparameters for the template itself? For flows and classifiers trained on a million events, that is the expensive problem warned about in Alternative 3 of the research file. Placing the Bayesian treatment on the low-dimensional residual surface is precisely what makes the treatment affordable.

Proposed Experiments

The primary experiment is a null calibration study on the LHC Olympics R&D dataset (The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics). We run several hundred background-only Herwig pseudo-experiments with Pythia-trained templates at $\epsilon_B \in \{10^{-2}, 10^{-3}, 10^{-4}\}$. The reported quantities are the fraction of pseudo-experiments exceeding nominal 2σ and 3σ, a Kolmogorov–Smirnov distance between the null Z distribution and a standard normal, and the coverage of the $\sigma_{sys}$ interval for the true residual.

The decisive prediction is as follows. The maximum-likelihood Gaussian process, and possibly the scalar $\delta_{sys}$, will exceed nominal false-positive rates at $10^{-4}$. The marginalized Gaussian process will not. The gap between them should track the size of the between-model term.

The second experiment injects the $W' \to XY$ signal over a range of $S/B$ and compares median $Z$ across all methods. This settles the sensitivity tension: marginalization should sit between the scalar method and the worst-case bound, and closer to the scalar method.

The third experiment is a blinded check on Black Box 1, which contains an injected signal, and Black Box 2, which is background only. Each method's $Z$ at a pre-registered working point is reported once.

The result we are after is a residual systematic that is honest at the tightest cuts, at a modest cost in sensitivity. If the scalar or inflated-scalar baseline is already calibrated everywhere, that negative result is worth publishing too: it would show that current practice is adequate in this regime.