Precision experiments now collect so many events that counting noise barely matters. What limits them are residual systematic uncertainties: imperfect knowledge of energy scales, background rates, and detector response. In an analysis these are handled as nuisance parameters, which are fit to the data together with the quantity of interest. A recent line of work trains neural networks to compress collision events into summaries that directly shrink the final error bar after this fit. This note starts from one observation. The matrix that sets how badly the nuisances inflate the error bar is the curvature of the likelihood in the nuisance directions. The same matrix appears inverted inside the gradient used to train the network. When a nuisance is weakly constrained and the likelihood goes flat in that direction, both the physics result and the optimization degrade together, and for the same reason.
The optimization literature has a vocabulary for this. It studies "bilevel" problems, in which an outer objective depends on the solution of an inner fit, and it shows that convergence depends on how convex the inner fit is. We propose to treat systematics-aware training as a bilevel problem whose inner fit is the actual nuisance profile, and to use that vocabulary to measure and control it. The practical claim is narrow and testable. Asimov-point Fisher objectives are what current methods optimize (the Asimov dataset is the idealized dataset whose counts equal their expected values). They are exact only when the profile is locally quadratic. Replacing them with a warm-started explicit profile fit should help where this approximation breaks, which is under shape nuisances and many correlated nuisances. It should also make training stability predictable from a measurable condition number.
Core Claim
Write $\mu$ for the parameter of interest, $\nu$ for the nuisance vector, and $I$ for the Fisher information of the binned likelihood built from the network summary $f_\theta$. The Fisher information is the expected curvature of the negative log-likelihood. The profiled variance of $\mu$ is the inverse of a Schur complement:
$$\sigma_\mu^2 = \left( I_{\mu\mu} - I_{\mu\nu}\, I_{\nu\nu}^{-1}\, I_{\nu\mu} \right)^{-1}.$$
Here $I_{\nu\nu}$ is the nuisance block, which includes the Gaussian constraint terms from auxiliary calibrations, and $I_{\mu\nu}$ is the coupling between signal and nuisances. In bilevel language, the inner problem is $\nu^*(\theta) = \arg\min_\nu \mathrm{NLL}(\mu, \nu; f_\theta)$. The hypergradient, meaning the gradient of the outer objective with respect to the network weights, contains $[\nabla^2_{\nu\nu}\mathrm{NLL}]^{-1}$. That is $I_{\nu\nu}^{-1}$ at the Asimov point.
The bilevel analyses of Bilevel Optimization: Convergence Analysis and Enhanced Design show that complexity scales with polynomial powers of the inner condition number $\kappa$. Their stochastic analyses also show how inner-solve error is amplified by $\kappa$ into hypergradient error. Taken together, the smallest eigenvalue of $I_{\nu\nu}$ is the direction where residual systematics are largest and also where outer gradients are least smooth and noisiest.
A structural fact sharpens this. Suppose expected bin counts are affine in the normalization nuisances, and constraints are Gaussian with widths $\sigma_j$. Then the Poisson negative log-likelihood is convex in those nuisances, and the constraint terms make it at least $1/\sigma_{\max}^2$-strongly convex, whatever the network does. This convexity is lost if normalizations are parameterized log-normally, and our experiments use both parameterizations.
Shape nuisances implemented by spline morphing, as in Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data, carry no such guarantee. Their data curvature can be negative and can cancel the prior curvature. Normalization nuisances therefore sit in the well-posed regime that classical bilevel theory assumes. Shape nuisances are where the regime studied by On Bilevel Optimization without Lower-level Strong Convexity becomes physically real. That paper proves first-order methods can stall on general non-convex-convex problems and recovers tractability under the Polyak-Łojasiewicz (PL) condition. The PL condition states that gradient size bounds the gap to the optimum, so every stationary point is global.
Relation to Prior Work
INFERNO, as applied by Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data, supplies the pipeline we reuse: softmax binning, spline morphing for the Jet Energy Scale (JES), and the inverse-Fisher loss. It is also the strongest competitor. It never solves the inner profile. It evaluates curvature at the Asimov point with one shape nuisance and gives no account of its own optimization behavior. We differ in three ways: we solve the profile explicitly on pseudo-datasets, we optimize the width of the actual profile-likelihood interval rather than its quadratic proxy, and we track $\kappa$ through training.
Adversarial pivoting, from Learning to Pivot with Adversarial Networks, enforces independence of the summary from nuisances through an unconstrained minimax game with a hand-tuned weight $\lambda$. It motivates this work by showing the instability and accuracy costs of forcing invariance. We do not seek invariance at all. We let the profile absorb nuisance dependence and penalize only what survives it.
Nuisance-Aware Prediction Sets (NAPS), from Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference, give post-hoc coverage guarantees under nuisance shifts. We use this as an evaluation standard: our intervals must cover on toys. It is not a training method.
The bilevel algorithms supply the method. A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse, A framework for bilevel optimization that enables stochastic and global variance reduction algorithms, and A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization contribute the idea of warm-starting the inner and linear-system variables across outer steps. Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis contributes step sizes that adapt without knowing curvature constants in advance.
One constraint shapes the design. Nuisance dimension here is tens, not millions. Hessian-free estimators such as those in On the Convergence Theory for Hessian-Free Bilevel Algorithms and Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimization solve a cost problem this setting does not have. We do not use them. Exact small-matrix solves are cheap, and the difficulty is curvature, not memory.
Research Program
The first component is a diagnostic study, and it is useful even if the method fails. We will take the $t\bar t$ $\tau$+jets setup from the INFERNO paper and extend it from one JES nuisance to a realistic set. The set will include background normalizations, and it will include additional shape shifts built by varying object energy scales in the simulated samples. The exact shape nuisances available depend on what the open simulation permits, and we will report which ones we could construct. For summaries trained by binary cross-entropy (BCE, the standard classification loss), by INFERNO, and by our method, we will record several quantities along training: the spectrum of $I_{\nu\nu}$, an empirical PL modulus of the profile on pseudo-datasets, and the gap between the Asimov-Fisher width and the width from a full profile scan. The hypotheses are specific. First, the Asimov-Fisher error concentrates in shape directions. Second, gradient-norm spikes and seed-to-seed divergence of INFERNO training line up with dips in $\lambda_{\min}(I_{\nu\nu})$.
The second component is the method, which we call profiled bilevel training. At each outer step, the inner nuisance fit is advanced by a few damped Newton steps from the previous step's solution. The damping is floored at the constraint curvature $1/\sigma_j^2$, which is a certified strong-convexity modulus for the normalization block. The inner fit runs on the Asimov set and on a small bank of Poisson pseudo-datasets, which exposes non-Gaussian profile shape.
The outer objective is the profiled interval width. It is obtained from where the profile crosses one unit in negative log-likelihood, found by a short root solve, and its gradient comes from the implicit function theorem. The step size is set from the monitored $\kappa$. We will also prove a modest result: for the normalization-only sub-problem, the explicit inner fit inherits linear inner convergence and a bounded hypergradient error from the constraint floor. For shape nuisances we report measured PL moduli rather than claim guarantees.
Baselines
The naive reference is BCE training on the same features and events, followed by the same profile-likelihood fit. INFERNO reports that this reference gives a signal-strength uncertainty of roughly 10–11%, a JES posterior uncertainty of 0.905, and a correlation of −0.42 between signal strength and JES (Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data).
The strongest existing method is INFERNO itself. It is the strongest because it already optimizes the physically relevant quantity, the profiled variance, rather than a proxy for it. On the single-JES Asimov setup it reports a signal-strength uncertainty of about 9%, a JES uncertainty of 0.602, and a correlation of −0.10 (same paper). We will first reproduce these numbers. Only then will we run INFERNO on the expanded nuisance set with the same network, events, number of optimizer steps, and a hyperparameter search of equal size to ours. Beating it means a smaller profiled interval on $\mu$, with toy coverage within Monte Carlo error of nominal, on the expanded set. We expect to tie it on the single-JES configuration. A tie there is the predicted outcome and counts as consistency, not a win.
Adversarial pivoting is a third baseline, with $\lambda$ scanned over a grid given the same total compute. Its reported W-tagging result is an Approximate Median Significance (AMS) of about 7.8 at $\lambda=10$ against about 7.0 for a standard classifier (Learning to Pivot with Adversarial Networks). We will not compare directly to that number, because it comes from a different dataset. We compare only within our setup.
Two ablations isolate the key ingredients. The first keeps our warm-started profile but uses the Asimov Fisher width as the outer objective. This separates the gain from the non-quadratic objective from the gain from explicit profiling. The second is INFERNO with a Tikhonov-damped Fisher block, which separates "handling the curvature properly" from "just regularizing it."
Why not just…?
Why not just run INFERNO with more nuisances? This is the main threat. INFERNO's Schur-complement loss is already implicit differentiation at a single point, so where profiles are close to quadratic it should match us. The diagnostic study settles the question. If the measured Asimov-versus-scan width gap stays small for every nuisance and INFERNO trains stably, then the explicit profile is unnecessary. In that case we would report the conditioning analysis as the result, and the method would add nothing.
Why not just damp or pseudo-invert the Fisher matrix when it goes singular? Damping fixes the numerics but changes the answer. Adding $\epsilon I$ to $I_{\nu\nu}$ is equivalent to tightening every constraint, which underestimates the systematic uncertainty. The damped-INFERNO ablation will show whether it produces falsely small intervals that under-cover on toys.
Why not just train with BCE and profile afterwards? The INFERNO paper already shows that BCE summaries leave uncertainty on the table (roughly 10–11% versus 9%). We keep BCE only as the reference point.
Why not just use adversarial pivoting? Pivoting removes nuisance dependence that the profile fit could otherwise absorb. When nuisance and signal are coupled, which is the case here through $I_{\mu\nu}$, it also throws away discriminating information. The within-setup comparison at matched compute will measure this cost directly.
Why not just use a Hessian-free bilevel solver? With tens of nuisances, forming and solving the nuisance Hessian costs almost nothing next to the network forward pass. Hessian-free estimation would add variance without any saving, so we do not use it.
Proposed Experiments
The main experiment uses the expanded-nuisance CMS Open Data $t\bar t$ analysis. We will report profiled $\mu$ interval width, toy coverage, width reduction attributable to each nuisance class, and seed-to-seed variance of the final width. The decisive check is a pattern across nuisance classes. On normalization-only configurations, our method should roughly match INFERNO. It should beat INFERNO specifically on configurations dominated by shape nuisances, and the size of the gain should track the measured quadratic-approximation gap. A gain that appears uniformly, including in normalization-only settings, would suggest a tuning artifact rather than the proposed mechanism.
The second decisive check is controlled. We extend the synthetic 2D benchmark of Learning to Pivot with Adversarial Networks with a tunable morphing nonlinearity and a tunable constraint width. This lets us dial $\lambda_{\min}(I_{\nu\nu})$ continuously. The prediction is that INFERNO's training failure rate and gradient variance rise as the curvature floor shrinks, while ours stay bounded down to where data curvature cancels the prior. The PL diagnostic should mark that boundary. Finally, we will apply the trained summaries to the real Open Data events and check consistency with the published measurement, as the INFERNO paper does.
References
- Bilevel Optimization: Convergence Analysis and Enhanced Design
- Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data
- On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysis
- Learning to Pivot with Adversarial Networks
- Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference
- A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse
- A framework for bilevel optimization that enables stochastic and global variance reduction algorithms
- A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization
- Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis
- On the Convergence Theory for Hessian-Free Bilevel Algorithms
- Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimization