Non-Linear Residual Experts for Constraining Detector Mismodeling

When a particle-physics experiment collects enough data, its final error bar is set by systematic uncertainty, not by statistics. Systematic uncertainty here means the leftover disagreement between the detector simulation and the real detector. A common way to handle it is to model control-sample data as "simulation, plus a small correction applied to some fraction of events." The size of that correction becomes the systematic error bar. The correction usually has a very simple form, such as a constant energy offset with some extra Gaussian smearing.

This note argues that this simple form has a hidden statistical cost. Recent theory on mixture-of-experts models (models that blend several sub-models, with a "gate" deciding how much each one contributes to each input) shows a specific failure when a small learnable component is added to a fixed base model and that component is linear with Gaussian noise. The component's offset and its variance become algebraically entangled. As a result, the fraction of affected events is estimated at rate $n^{-1/4}$ or slower instead of $n^{-1/2}$. Making the component non-linear removes the entanglement. We propose to carry this result into detector calibration and test the practical claim: if the residual correction uses a non-linear (saturating) response curve and an independent sigmoid gate, the same control sample constrains the residual systematic much more tightly, and the propagated intervals on physics parameters still cover the truth at the nominal rate.

Core Claim

The residual-correction model we study is a contaminated mixture: a fixed simulation-derived response density plus a small, gated residual component. Here $y$ is a reconstructed quantity (for example the ratio of reconstructed to true jet momentum), $x$ are per-event features, $f_0$ is the fixed simulation response, $\lambda \in [0,1]$ is the overall size of the mismodeling, $g(x)=\mathrm{sigmoid}(w^\top x + c)$ is a gate giving the probability that an event is affected, and $h(x;a,b)$ and $\nu$ are the residual component's mean and variance:

$$p(y \mid x) = \big(1-\lambda g(x)\big)\, f_0(y \mid x) + \lambda g(x)\, \mathcal{N}\big(y \mid h(x;a,b),\, \nu\big).$$

The Gaussian density satisfies the heat-equation identity: its derivative with respect to the variance equals half its second derivative with respect to the mean. When the mean is linear, $h = a^\top x + b$, the derivative in $b$ is exactly the derivative in the mean. So the second derivative in the offset $b$ is a fixed multiple of the derivative in the variance $\nu$. This is the linear dependence that Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts identifies as the cause of slow minimax rates for a fixed base model plus a learnable "prompt" expert. That paper also shows that a non-linear expert restores the $n^{-1/2}$ rate for the mixing proportion.

With a saturating mean, $h = \tanh(a^\top x + b)$, the chain rule adds an input-dependent factor and an extra first-derivative term, so the dependence breaks. Saturating response curves are also physically natural for calorimeter non-linearities.

The claim has three parts.

  1. The standard linear offset-plus-smearing residual constrains $\lambda$ at a visibly sub-parametric rate on realistic jet data, especially when $\lambda$ is small.
  2. A non-linear residual with an independent sigmoid gate reaches close to $n^{-1/2}$.
  3. This difference survives propagation. At fixed coverage, intervals on a downstream physics parameter are narrower for the same control-sample size.

What is new is not "use a mixture of experts in physics." It is pointing to a specific, provable degeneracy in a residual-systematic parameterization that physicists already use, and testing whether the theoretical fix has practical consequences.

Relation to Prior Work

Four lines of theory supply the mechanism. On Least Square Estimation in Softmax Gating Mixture of Experts proves that linear or polynomial experts fall to logarithmic parameter rates, while "strongly identifiable" experts (sigmoid, tanh, GELU) reach $n^{-1/2}$. Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts shows that the sum-to-one constraint of softmax gating makes experts compete in a zero-sum way, and that independent sigmoid gates avoid this. The contaminated-mixture paper above gives the closest structural match to our setting. Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? shows the same kind of decoupling for temperature parameters.

All of this work assumes the data truly come from the fitted mixture, and it is tested only on synthetic regression. Those papers themselves list misspecification as unresolved. Detector data are never exactly a two-component mixture, so we treat misspecification as a first-class experimental arm rather than assuming it away. The contaminated-mixture theory also uses a constant mixing weight. Our input-dependent sigmoid gate is an extension, and its rate must be checked rather than assumed.

The physics side supplies the setting and the competitors. Parameterized Machine Learning for High-Energy Physics feeds physics and nuisance parameters into a standard network $f(x,\theta)$, so one model interpolates across parameter values. It never asks whether the nuisance parameters are statistically well identified. Learning Uncertainties the Frequentist Way provides the jet-calibration testbed and per-jet resolution estimates, using monolithic networks with no residual decomposition. Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation makes likelihood-ratio estimators conservative, which we use as the "just widen the intervals" competitor. Direct Amortized Likelihood Ratio Estimation estimates ratios between pairs of parameter values directly.

These ratio-estimation methods address how well a network approximates a likelihood. Our proposal addresses something different: whether the residual model's own parameters are learnable at a fast rate.

Research Program

The work has three stages.

Stage 1: confirm the mechanism in our gated form. We extend the contaminated-mixture argument to the input-dependent sigmoid gate. We check whether the gate's parameters $(w,c)$ create a new degeneracy with $\lambda$; they plausibly do, because $\lambda$ multiplies the gate. We then fix the parameterization, for example by normalizing the average gate output to one so that $\lambda$ alone carries the overall scale. Alongside the proof, we replicate the synthetic rate experiments so that our estimators match the published slopes before touching physics data.

Stage 2: jet calibration. We use the simulated CMS dijet sample (5 million jets, $p_T \in [500,1000]$ GeV) from Learning Uncertainties the Frequentist Way. The response density $f_0$ is a normalizing-flow or Gaussian-Ansatz fit to the nominal simulation, and it is frozen. We then build pseudo-data by injecting a controlled mismodeling into a held-out split. The injected mismodeling is a flavor-like population of jets, selected by particle-composition features, whose response is shifted and smeared. Its size $\lambda^\ast$ ranges from $10^{-1}$ down to $10^{-3}$, and the shift is either linear or saturating in true $p_T$.

This makes the truth known, so we can measure estimation error in $\lambda$, $a$, $b$ and $\nu$ as the control-sample size $n$ varies from $10^3$ to $10^6$.

Stage 3: propagation to a physics parameter. The fitted residual is propagated into a downstream inference where the residual parameters act as constrained nuisances. One downstream case is the resonance mass on HEPMASS, inferred with a parameterized classifier in the style of Parameterized Machine Learning for High-Energy Physics. The other is the Weinberg angle benchmark from Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation, with the same style of injected response residual.

Baselines

Do nothing. Use the nominal simulation with no residual component, and assign a conventional Gaussian-prior nuisance on the energy scale. This shows how much bias and undercoverage the injected mismodeling causes when it is ignored. Any method worth adopting must reduce that bias.

Standard linear residual. This is what an analysis would actually do, and it is the strongest competitor. The residual is the same mixture as above but with a linear mean $h=a^\top x+b$ and a constant $\lambda$. It is fit by maximum likelihood, and its intervals come from profile-likelihood scans. Both this baseline and our model use the same frozen $f_0$, the same control samples, the same optimizer, and the same number of random restarts; both mixtures are non-convex, so restarts matter. The comparison is also run with the identical sigmoid gate attached to the linear residual, so that the only difference is the shape of the residual's mean.

On the published record, linear experts reach only $n^{-0.06}$ in parameter error, against $n^{-0.54}$ for ridge experts, in On Least Square Estimation in Softmax Gating Mixture of Experts. For the contaminated setting, no reported scalar result was found on Lacuna for Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts. The strongest monolithic calibration network on the dijet data, PFN-PID, is reported to improve jet resolution by about 15% over the CMS baseline in Learning Uncertainties the Frequentist Way. We reuse that network as the source of $f_0$, so our comparison sits on top of the best available calibration rather than a weak one.

Beating the linear residual means three things at once:

  • a fitted error-versus-$n$ slope for $\lambda$ that is distinguishably steeper (confidence intervals on the slopes do not overlap);
  • equal or better empirical coverage at 68% and 95%;
  • narrower downstream intervals at matched coverage.

Parameterized ratio estimators. These are the Baldi-style network and Balanced Neural Ratio Estimation (BNRE), given the residual parameters as extra inputs with the same simulation budget. They are trained with the published default balancing weight of 100, and given the same hyperparameter search budget as our model.

Gating ablations. Two ablations isolate the key ingredient. The first swaps sigmoid gating for a two-component softmax gate. The second keeps the non-linear expert but removes the gate, using a constant $\lambda$. Together they separate the expert-shape effect from the gate effect.

Why not just…?

Why not just feed nuisance parameters into one big network, as in parameterized machine learning? A network $f(x,\theta)$ can interpolate well. But it still needs some parametric model of what the residual is, and the identifiability of that model is a property of the statistics, not of the network's capacity. If the residual is written as offset plus smearing, the heat-equation coupling is present no matter how expressive the classifier is. The Stage 3 comparison tests this directly: the same parameterized network is used downstream with either residual model upstream.

Why not just make the estimator conservative with BNRE? Conservative estimators repair coverage by widening intervals. Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation reports positive coverage area under the curve across simulation budgets, but no reported scalar result was found on Lacuna for the Weinberg task. If slow residual estimation is the real problem, BNRE should reach coverage only at the cost of interval width that grows relative to our model as $n$ increases. If BNRE matches both coverage and width, that would mean the rate gap does not matter in practice, and we would report that outcome.

Why not just fix the smearing variance from an external calibration? This is a real threat. With $\nu$ known, the heat-equation coupling disappears, and the linear model might recover fast rates without any non-linearity. We include an arm where $\nu$ is fixed: once at its true value, and once at a value off by 10–30%. Our expectation is that a correct $\nu$ closes much of the gap, while a mis-set $\nu$ biases $\lambda$. If the correctly fixed variance closes the gap completely, the honest recommendation becomes "pin the variance when you can," and our method is needed only where the variance cannot be pinned.

Why not just reweight simulation to data with a classifier? Classifier reweighting can absorb any mismodeling, but it gives no small set of interpretable parameters that can enter a likelihood as nuisances. Its uncertainty must come from bootstrapping. We run it with the same data and report bootstrap coverage and downstream interval width. It may well win under severe misspecification, which is exactly where the parametric theory is silent.

Proposed Experiments and Decisive Checks

Check 1: rate replication. On the synthetic regression setup (32-dimensional inputs, $n$ from $10^3$ to $10^5$) we must reproduce the published slopes before trusting our code: sigmoid gating at about $n^{-0.51}$ versus softmax at about $n^{-0.24}$ for ReLU experts, from Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts.

Check 2: the central test on dijet pseudo-data. We plot log estimation error of $\lambda$ against log $n$ for the linear and non-linear residuals, across injected sizes $\lambda^\ast$. The prediction is a slope near $-1/4$ or shallower for the linear residual and near $-1/2$ for the non-linear one, with the gap widening as $\lambda^\ast$ shrinks. At $n=10^6$ the predicted error ratio between the two is roughly an order of magnitude; this follows directly from comparing $n^{-1/4}$ with $n^{-1/2}$. A gap that small or absent in practice would falsify the practical relevance of the theory.

Check 3: misspecification. We cross the generating residual (linear or saturating shift) with the fitted residual (linear or non-linear) and report bias and coverage in all four cells. The idea succeeds only if the non-linear fit to a linearly generated truth stays nearly unbiased in $\lambda$. If it does not, the gain is merely an artifact of matching the truth.

Check 4: downstream propagation. For the HEPMASS resonance mass and the Weinberg angle, we report empirical coverage at 68% and 95% and the median interval width, using 1,000 pseudo-experiments for each configuration. The result counts as a win only if the non-linear residual reaches nominal coverage with narrower intervals than the linear residual, BNRE, and classifier reweighting at the same control-sample size.