Physicists increasingly use neural networks to compare a hypothesis against data. A common tool is a density ratio, which says how much more likely an event is under one hypothesis than under another. To be trustworthy, the resulting measurement needs frequentist error bars, meaning intervals that contain the true value at their stated rate (68%, 95%, and so on) across repeated experiments. The wifi method of Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles offers a clean way to get such intervals. It freezes a set of trained networks $f_i$, fits a linear combination of them, $\log \tilde r(x\mid w)=\sum_i w_i f_i(x)$, on held-out data, and reads the uncertainty on the weights $w_i$ off an analytic covariance formula, with no bootstrapping. The catch, which the paper states itself, is that the intervals are only correct if the true ratio lies in the span of the frozen networks.
That assumption is most likely to fail in rare regions of the input space: unusual jet topologies, sparse tracker hits, or sub-detector channels that seldom fire. The networks saw very few examples there, which is the same "rare features are under-trained" problem documented in the sparse-optimization literature. The resulting error is a bias, not a variance, so the error bars do not grow to cover it. We propose a simple repair that turns this uncovered bias into honest, measured variance. Alongside the frozen networks, we add a small set of low-capacity "tail" basis functions, one per sparsely populated region of phase space, and fit their weights in the same linear step. Each added weight then carries a variance that scales with the inverse of the event count in its region. Where the data is thin, the interval widens in step with Poisson counting statistics instead of silently undercovering.
Core Claim
In neural density-ratio inference, coverage failure concentrates in regions with few training events. Augmenting the frozen basis with region-indicator functions restores nominal coverage at a predictable cost in interval width, roughly $1/\sqrt{N_k}$ for a region holding $N_k$ fit-sample events. Improving how the networks themselves are trained does not achieve this. That includes frequency-adaptive optimizers, tail oversampling, and larger ensembles, because those methods can shrink bias but cannot certify that it has been removed.
The idea borrows its logic from confidence-weighted learning. Confidence-weighted linear classification and Exact Convex Confidence-Weighted Learning give every feature its own variance, so rare features stay uncertain and are updated aggressively. Those papers use per-feature variance to steer online optimization. We use it differently: per-region weights become explicit statistical parameters whose analytic variance reports how little the data says about each tail. The augmented model is
$$\log \tilde r(x\mid w,v)=\sum_{i=1}^{M} w_i f_i(x)+\sum_{k=1}^{K} v_k\, g_k(x),$$
where $f_i$ are the $M$ frozen networks and $g_k(x)$ indicates (or smoothly localizes) membership in rare region $k$. The coefficient $v_k$ is a correction to the log-ratio in that region. Because the $g_k$ have nearly disjoint support, the information matrix is close to block-diagonal in $v$. The variance of $v_k$ therefore scales as $1/N_k$, by the same counting argument that gives Poisson relative error $1/\sqrt{N}$. That is where the width prediction comes from.
Research Program
We first build a diagnostic, then the repair. For the diagnostic, we take the wifi pipeline as published, run many independent pseudo-datasets with known ground truth, and record coverage of the inferred physical parameter, the quark fraction $\kappa$. We stratify this coverage by how densely the basis-training sample populated the region that drives the measurement. The hypothesis to confirm or refute is that coverage is nominal in the bulk and falls below nominal as region density drops.
The repair has three design choices. The first is how regions are defined. We bin a small number of physically meaningful, frequency-imbalanced observables, such as track multiplicity, the count of active sub-detector channels, and constituent multiplicity, and we flag bins whose basis-training count falls below a threshold. Regions are chosen on the basis-training split only, so the fit split is not used twice. The second is the form of $g_k$, either hard indicators or soft kernels. The third is how to keep the fit stable when $K$ is large. wifi already reports a small bias in high dimensions that needs a leading-order correction, so we cap $K$, merge adjacent sparse bins, and apply the same correction. We also confront the data-splitting cost that wifi imposes. The tail weights are fit on the same held-out split as $w$, so no further splitting is required.
Baselines
The naive reference is a single network, or a naive average of networks, used as a plug-in ratio. Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles reports that naive ensembles "often exhibited substantial bias and under-covered the true parameter values." We expect them to fail everywhere, and including them anchors the scale of the problem.
The strongest existing method is the unmodified wifi ensemble, run with the paper's symmetrized maximum-likelihood loss and its high-dimensional bias correction. Its reported convergence figure is that coverage reached the nominal 68% at ensemble size $M=16$, but that result comes from a 1D Gaussian toy. On the quark/gluon task, no scalar coverage number is reported on Lacuna; the summary says only that intervals "accurately captured the ground truth." We will regenerate this baseline ourselves on the same Pythia sample, with the same $M$, the same data splits and the same basis networks. The augmented method beats it if two things hold. It must keep coverage within Monte Carlo error of nominal in the lowest-density strata, where the baseline undercovers. Its median interval width in the bulk must also stay within a few percent of the baseline's.
The key-ingredient ablation keeps $K$ fixed but places the $g_k$ on randomly chosen high-density regions instead of rare ones. If coverage recovers equally well, the effect comes from extra parameters, not from targeting rare regions.
A second family of competitors fixes training rather than inference. One variant trains the $f_i$ with frequency-aware updates in the spirit of CowClip, which clips gradients per embedding column based on ID frequency, and of Bloc-AdaGrad, which keeps global decay from freezing rare parameters. A further variant oversamples tail events during basis training. Every variant gets the same optimizer-tuning budget as the baseline.
Why not just…?
Why not just train the basis networks with a frequency-adaptive optimizer? Prior work suggests standard optimizers do mishandle rare features. CowClip shows that global clipping starves rare IDs under large batches, and Green et al. (Fast and Adaptive Online Training of Feature-Rich Translation Models) needed AdaGrad combined with FOBOS regularization to handle dense and sparse features together. Better training should therefore reduce tail bias. It cannot make the error bars account for whatever bias remains. We test this directly: if optimizer-fixed bases alone restore tail coverage, the augmentation is unnecessary and we will report that. Our prediction is that they narrow the gap without closing it, and that combining both works best.
Why not just oversample or reweight the tail during basis training? This is the most serious threat, because it attacks the data starvation itself. It reallocates network capacity, but it adds no new information. Its residual bias is also invisible to the sandwich covariance, the analytic covariance formula wifi uses for the weights. If oversampled bases achieve nominal tail coverage, our claim fails. The stratified coverage experiment settles this.
Why not just bootstrap the whole pipeline or enlarge the ensemble? Retraining the networks on resamples captures variance from training. It does not capture systematic misspecification that every resample shares. For the same reason, more networks trained the same way share the same blind spots. Bootstrapping is also expensive; RESuM shows how quickly simulation-heavy budgets explode in rare-event physics. We include a 100-replicate full bootstrap and an $M=64$ ensemble as references.
Proposed Experiments
The first testbed is a 1D heavy-tailed toy that extends wifi's Gaussian toy. There, the true ratio is known analytically, and tail density can be dialed down. The main testbed is the Pythia 8.226 quark/gluon sample of 2 million jets with $p_T\in[500,550]$ GeV. On it we build quark/gluon mixtures with known $\kappa$, and in some mixtures the composition difference is concentrated in rare-multiplicity jets. To probe genuinely sparse detector channels, we repeat the analysis on the CMS Open Data detector images from End-to-End Jet Classification of Quarks and Gluons with the CMS Open Data. In that dataset, track images are much sparser than calorimeter images, and the reported classifier reaches ROC AUC 0.8076 with combined inputs. We will use that architecture for the $f_i$ and check that our trained networks reach comparable AUC before running any inference study.
The metrics are empirical coverage at 68% and 95% over at least 1,000 pseudo-experiments per stratum, bias in $\kappa$, and median interval width. All three are reported as functions of $N_k$.
The decisive check is a single plot of coverage against region training density for every method. The proposal succeeds if three conditions hold together: the wifi baseline drops below nominal at low $N_k$, the augmented model stays nominal, and the augmented interval width tracks $1/\sqrt{N_k}$. It fails if tail oversampling or frequency-adaptive training produces the same flat coverage curve without the indicator bases.
References
- Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles
- Confidence-weighted linear classification
- Exact Convex Confidence-Weighted Learning
- CowClip: Reducing CTR Prediction Model Training Time from 12 Hours to 10 Minutes on 1 GPU
- Comparison of scheduling methods for the learning rate of neural network language models (Modèles de langue neuronaux: une comparaison de plusieurs stratégies d'apprentissage) [in French]
- Fast and Adaptive Online Training of Feature-Rich Translation Models
- RESuM: A Rare Event Surrogate Model for Physics Detector Design
- End-to-End Jet Classification of Quarks and Gluons with the CMS Open Data