Precision physics measurements are now limited less by how many events are recorded than by the mismatch between the simulator and the real detector. Current machine-learning inference methods handle this mismatch only when someone has written it down in advance as a knob, or "nuisance parameter," in the simulation. Effects nobody anticipated get no error bar at all. Number theorists face a similar situation when a foundational hypothesis is unproven. They state the hypothesis explicitly, prove what follows from it, and treat any downstream contradiction as evidence against the hypothesis. We propose to do the same for unmodeled systematics. Instead of a single interval, an analysis would report an interval that is valid if the unmodeled simulator–reality discrepancy is no larger than a stated size $\varepsilon$. It would also report a data-driven test, run in signal-free control regions, that can reject values of $\varepsilon$ that are too small.
The technical core is a bias bound for density-ratio estimators built from a fixed set of neural basis functions. Its key property is that only the part of the discrepancy lying outside the span of the basis, weighted by how sensitive the physics parameter is to it, can bias the result. Discrepancies inside the span are absorbed by refitting. This gives error bars that are tighter than the generic "inflate by the control-region non-closure" practice, while keeping the assumption they rest on visible and testable.
Core Claim
The method builds on the $w_i f_i$ ensemble of Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles. That work trains $M$ networks $f_i$, freezes them, fits linear weights $w_i$ on independent data, and propagates a sandwich covariance to a physics parameter such as the quark fraction $\kappa$. Its coverage guarantee holds only if the true log density ratio lies in $\text{span}\{f_1,\dots,f_M\}$. The authors acknowledge that the guarantee fails otherwise, but they do not quantify how badly.
Suppose the true log ratio equals an in-span part plus a residual $g$ orthogonal to the span, with $\|g\|_{L^2(p)} \le \varepsilon$, where $p$ is the simulated reference density. Then, to first order, the shift in the estimated parameter $\hat\theta$ obeys
$$|\,\mathbb{E}\hat\theta - \theta\,| \;\le\; \varepsilon \,\|\psi_\perp\|_{L^2(p)},$$
where $\psi$ is the influence function of $\theta$ with respect to the log ratio, and $\psi_\perp$ is its component orthogonal to the basis span. The bound follows from linearizing the M-estimator's pseudo-true value in $g$ and applying Cauchy–Schwarz. The conditional interval is the usual sandwich interval widened by this bias term. Because $\psi_\perp$ can be estimated from simulation, the widening is computable without pseudo-experiments.
The unproven premise is made explicit as a transfer conjecture. It states that the orthogonal discrepancy in the signal region is at most a stated factor times the discrepancy measured in control regions. Control-region data give a lower confidence limit on $\varepsilon$. Any reported interval must use an $\varepsilon$ that this test does not reject. This mirrors conditional proofs such as the GRH-dependent error terms in The first moment of quadratic twists of modular $L$-functions: the dependence on the hypothesis is stated rather than hidden, and a falsified consequence counts against the hypothesis.
Relation to Prior Work
Three works motivate the proposal. The wifi paper supplies the estimator and the misspecification gap it leaves open. The FAIR Universe challenge (FAIR Universe HiggsML Uncertainty Challenge Competition, FAIR Universe HiggsML Uncertainty Dataset and Competition) benchmarks signal-strength inference under six parameterized nuisances. Its organizers state that current methods address only "known unknowns." We do not propose a better nuisance model. We propose an error term for whatever the nuisance model leaves out.
Likelihood-free frequentist inference (Likelihood-Free Frequentist Inference) constrains the design in two ways. It shows that asymptotic shortcuts such as Wilks' theorem can fail, so we check the first-order bound empirically rather than trusting it. It also relies on a smoothness assumption across nuisance space, which we inherit: our linearization fails for discontinuous discrepancies, and one experiment probes exactly this.
TaylorNet (TaylorNet: A Taylor-Driven Generic Neural Architecture) is the file's machine-learning example of a result conditional on a numerically verified but unproven conjecture. We borrow its practice of explicit conditional statements, not its architecture.
Research Program
The work has three parts. The first derives the bound, including second-order corrections and a finite-sample estimator of $\|\psi_\perp\|$ with its own uncertainty. We would also characterize when the linearization is tight: small $\varepsilon$ and smooth $g$.
The second part builds the control-region test. It estimates the orthogonal part of the data-versus-simulation log ratio in signal-free regions, using a held-out classifier projected off the frozen basis, and returns a one-sided lower limit on $\varepsilon$.
The third part is a "hidden nuisance" protocol that turns known unknowns into controlled unknown unknowns. In FAIR Universe, the analyst models only five of the six nuisances. The sixth, for example the jet energy scale, is applied secretly to pseudo-data. The analyst's conditional interval at the control-region-selected $\varepsilon$ is then scored for coverage. Rotating which nuisance is hidden gives six distinct misspecification scenarios with known ground truth.
Baselines
The naive reference is the unmodified wifi sandwich interval applied under the hidden nuisance. We expect it to under-cover. The wifi authors already report that naive averaged ensembles are biased, and that wifi reaches nominal 68% coverage on their Gaussian toy at $M=16$ only when the model is correctly specified. The standard XGBoost challenge baseline, which ignores systematics, gives a second naive point.
The strongest existing method is the HEPHY unbinned surrogate (GOLLUM), which reported a Quantile Score of 0.878 on FAIR Universe with all six nuisances modeled. Contrastive normalizing flows (IBRAHIME) scored 0.823. We will run the best reimplementation available to us of each under the same hidden-nuisance protocol: same training events, same five modeled nuisances, same hyperparameter-search budget as our method.
Beating these baselines means reaching a higher Quantile Score than each under hidden nuisances, with empirical coverage of the 68.27% interval not below nominal. The fully modeled 0.878 serves as an oracle ceiling, not a target.
The practitioner baseline is closure-test inflation. It adds the control-region data/simulation non-closure to the interval as a symmetric systematic, propagated with the full sensitivity instead of the orthogonal part. The key ablation replaces $\|\psi_\perp\|$ with $\|\psi\|$ and removes the projection. This isolates whether restricting to out-of-span discrepancies is what buys the tightness. The expensive reference is a Neyman construction with pseudo-experiments over adversarial perturbations inside the $\varepsilon$-ball. It is feasible on the quark/gluon set and is used to validate the linear bound.
Why not just…?
Why not just add a generic catch-all nuisance parameter to the simulation? A single extra template only covers the direction it encodes. If the hidden jet-energy-scale shift is not aligned with it, coverage fails. The hidden-nuisance rotation tests this directly by pairing each hidden nuisance with a fixed generic template and measuring coverage. We expect failures for at least some rotations.
Why not just inflate by the control-region non-closure, as analyses already do? This is the real threat. It also needs a transfer assumption and costs nothing extra. Our claim is only that the orthogonal-projection weighting gives narrower intervals at equal coverage. The $\|\psi\|$ ablation is exactly this comparison. If widths are not meaningfully narrower, the contribution shrinks to making the transfer assumption explicit and testable, and we would report it that way.
Why not just bootstrap everything? Both the wifi and LF2I papers document the cost. LF2I reports hours per observation for the 51,200-dimensional GEANT4 calorimeter task, versus seconds with amortized calibration. Bootstrapping also does not help unless perturbations beyond the simulator are drawn, and choosing them requires the same $\varepsilon$-ball assumption. We use the bootstrap only as a validation reference.
Why not just use more basis functions? A larger $M$ shrinks the out-of-span error only for discrepancies the simulator can produce. Real mis-modeling is absent from training by construction. We will vary $M$ and show that hidden-nuisance bias does not vanish.
Proposed Experiments
The first experiment is the wifi Gaussian toy with injected orthogonal perturbations of known norm. It checks that the bound is tight to first order and identifies where second-order terms matter.
The second uses the Pythia 8.226 quark/gluon jets, 2 million jets with $p_T$ in 500–550 GeV. Here we inject reweighting perturbations to the gluon template and measure coverage, bias, and width on $\kappa$ against the adversarial Neyman reference. The file reports no numerical coverage figure for wifi on this set, only that it "empirically satisfies" coverage, so we will regenerate that baseline ourselves.
The third is the six-way hidden-nuisance rotation on FAIR Universe, scored with the challenge's Quantile Score and raw coverage.
The decisive check is the combined result on FAIR Universe. Conditional intervals at the control-region-selected $\varepsilon$ must reach nominal coverage in most rotations where the naive and GOLLUM-style baselines under-cover. They must also be narrower than closure-test inflation. A deliberately non-smooth hidden perturbation, a step in detector response, tests where the linear bound breaks. The control-region test should flag such cases by rejecting the transfer conjecture rather than silently under-covering.
References
- Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles
- The first moment of quadratic twists of modular $L$-functions
- FAIR Universe HiggsML Uncertainty Challenge Competition
- FAIR Universe HiggsML Uncertainty Dataset and Competition
- Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning in Simulation and Uncertainty Quantification
- TaylorNet: A Taylor-Driven Generic Neural Architecture