When a rare signal is buried under a huge background, physicists rarely use a classifier only to make yes/no decisions. They feed each event's classifier score into an unbinned extended maximum likelihood fit. This is a statistical model that evaluates continuous signal and background densities at every event's exact score and treats the total event count as a Poisson random number. The output is how many signal events are present, with an honest error bar. The machine learning literature on minority class oversampling evaluates its methods with classification metrics such as AUC, F1 and G-mean. Oversampling here means duplicating or synthesizing rare-class examples so the classifier sees them more often. Its own literature disagrees about how far to oversample: one study says go to 1:1 balance, another finds 0.5:1 works better.
This note argues that, for rare-signal counting, the disagreement comes from measuring the wrong thing. What matters downstream is the Fisher information the score carries about the signal yield. When signal is rare, that information is dominated by the small, nearly background-free tail of the score distribution, which AUC and F1 barely weight. We propose to measure oversampling methods and ratios by the precision of the extended-fit yield. We will test whether this ranking disagrees with classification metrics. We will also build a simple procedure that picks the oversampling ratio, and decides where synthetic samples should be placed, by maximizing validation-set yield information.
Core Claim
Take a classifier with score $y$. Let $f_s(y)$ and $f_b(y)$ be its score densities under signal and background, estimated from held-out data that was never oversampled. Let $s$ and $b$ be the expected signal and background yields. The extended likelihood of $N$ observed events is $e^{-(s+b)}\prod_{i=1}^{N}\left(s f_s(y_i)+b f_b(y_i)\right)$. The expected information about $s$ is
$$I_{ss}=\int \frac{f_s(y)^2}{s f_s(y)+b f_b(y)}\,dy \;\approx\; \frac{1}{b}\left(1+\chi^2(f_s\,\|\,f_b)\right)\quad (b\gg s),$$
where $\chi^2$ is the chi-square divergence between the two score densities. The approximation follows from dropping $s f_s$ in the denominator. The divergence grows with $(f_s/f_b)^2$, so it rewards scores that push even a small fraction of signal into regions where background is nearly absent. AUC averages over all signal-background pairs and is largely insensitive to that tail.
Three things follow. First, oversampling cannot bias the fit if the templates $f_s, f_b$ come from untouched held-out samples. The "oversampling breaks Poisson statistics" worry raised in the research file applies only to the mistake of fitting oversampled data, and we treat it as a bookkeeping rule rather than a research problem. Second, with unlimited model capacity, oversampling is irrelevant, because any monotone transform of the likelihood ratio gives the same $I_{ss}$. Third, oversampling should therefore matter only through finite capacity and finite signal statistics. It changes which part of score space the network spends its capacity resolving. Our hypothesis is that the ratio maximizing $I_{ss}$ differs systematically from the ratio maximizing AUC or F1. We further hypothesize that generative oversamplers help only when their synthetic points land in the high-purity tail rather than in the overlap region.
Relation to Prior Work
The method side builds on standard unbinned fits. The CLEO $\tau\to\mu\gamma$ search (Update of the search for the neutrinoless decay τ→μγ) used a two-dimensional extended fit over energy and mass differences. The charmless $B$ analysis (Observation of B+→ωK+) folded a Fisher event-shape discriminant into a multi-variable unbinned fit. Both used linear or hand-built discriminants. We replace these with a deep classifier and ask how its training distribution should be set. Signal estimation in on/off measurements including event-by-event variables shows that single-event likelihoods beat hard cuts by 15–20% in precision at 10% signal-to-noise. It also notes that its simulated densities degrade when simulation is scarce. That scarcity is the regime we target, and it motivates the work.
On the deep learning side, Searching for exotic particles in high-energy physics with deep learning trained deep networks on the HIGGS and SUSY benchmarks without imbalance handling. It reported classification significance, not fit-based yield precision. We reuse its data and architecture and change both the training distribution and the evaluation. Publishing unbinned differential cross section results is the closest bridge between ML and unbinned physics. It uses classifier reweighting and normalizing flows for unfolding, but it neither treats imbalance nor runs extended fits. Our question is narrower: yield extraction, not unfolding.
The imbalance literature supplies the interventions and the puzzle we try to resolve. A systematic study of the class imbalance problem in convolutional neural networks found that replication oversampling dominates undersampling and thresholding, and recommended 1:1 balance, judged by ROC AUC. CASIA@SMM4H'22 found 0.5:1 beats 1:1 by F1. We predict that both answers can shift under a yield-precision criterion, and that the optimum depends on the deployment signal fraction. SMOTE interpolates between neighboring minority points. Oversampling for Imbalanced Data via Optimal Transport maps prior samples onto the minority manifold and shifts them toward the decision boundary. SOS: Score-based Oversampling for Tabular Data generates minority rows with a diffusion model. These are the generators we compare.
One point needs flagging. Both the optimal-transport method and SOS deliberately concentrate synthetic samples near the class boundary. Our analysis predicts this is the wrong place for rare-signal counting, so these methods are the most direct test of the claim. Revive Re-weighting in Imbalanced Learning by Density Ratio Estimation shows that dynamic reweighting beats static weights on long-tailed images, and it serves as the strongest loss-side competitor. Sampling a Longer Life shows that sampled binary classifiers beat one-class models when the majority class is multi-modal. Physics backgrounds with several production channels are multi-modal, so this result constrains our design: we stay with binary classifiers and do not switch to anomaly detection.
Research Program
The first component is an evaluation harness for the HIGGS and SUSY benchmarks from Baldi et al. These datasets are roughly balanced as released. We therefore induce imbalance by subsampling signal in the training set to signal-to-background ratios of 1:100, 1:1000 and 1:10000. This mimics the realistic case where signal simulation is expensive or comes from a small control sample. We keep a disjoint, never-oversampled template set and a large pseudo-experiment pool.
For each trained classifier we build $f_s, f_b$ from the template set. To keep these estimates stable in the sparse tail, we use kernel density estimation on a logit-transformed score and cross-fitting across folds. We then compute $I_{ss}$ analytically. We also run thousands of toy extended fits with injected signal at realistic deployment fractions of 0.1%, 1% and 5%, with the background yield floated. From the toys we record yield root-mean-square error, bias, pull width and 90% interval coverage. Template uncertainty is propagated by bootstrapping the template set.
The second component is an analysis of where capacity goes. We will compare score densities across oversampling ratios and generators, and decompose $I_{ss}$ by score quantile. The goal is to check directly that gains or losses come from the high-purity tail, as the $\chi^2$ form predicts. For each generator we also measure where synthetic points fall: on the classifier's eventual purity axis, and relative to physical boundaries in feature space. One example of such a boundary is a negative invariant mass produced by SMOTE interpolation.
The third component is a procedure we call information-matched oversampling. It has two parts. One selects the oversampling ratio by maximizing validation $I_{ss}$ at the expected deployment fraction rather than validation AUC. The other is a tail-targeted variant: synthetic signal from SOS or SMOTE is accepted only if a preliminary classifier assigns it to the top-purity region, which reverses the boundary-seeking bias of the optimal-transport method and SOS. To check that the finding is not physics-specific, we repeat the full study on the tabular benchmarks used by SOS, Shoppers and Default. There the analogous downstream task is estimating how many positives an unlabeled deployment batch contains, using the same extended likelihood.
Baselines
The naive reference is a network with the Baldi et al. architecture trained on the imbalanced data with plain cross-entropy and no intervention. Its scores feed the identical extended fit. Any method must beat this to justify itself. The closest published anchor on the balanced task is Baldi et al.'s deep network reaching 4.9σ significance on low-level HIGGS features, against 2.5σ for a shallow network. We use their architecture and training recipe as the starting point. However, no reported result was found on Lacuna for extended-fit yield precision on HIGGS or SUSY under induced imbalance, so that number must be established here.
The strongest data-level baseline is random oversampling to 1:1, as recommended by Buda et al. A second data-level baseline is ratio-tuned oversampling with the ratio chosen by validation AUC or F1, which is how a practitioner following Fu et al. would act. The strongest generative baseline is SOS. On Shoppers it reports weighted F1 of 0.675, against 0.648 for SMOTE and 0.601 for the identity baseline (SOS). We will reproduce those classification numbers before reporting yield precision on that dataset. SMOTE and the optimal-transport oversampler are included as the local-interpolation and boundary-shifting references. The strongest loss-side baseline is dynamic density-ratio reweighting from the Revive Re-weighting paper, together with static class weighting.
Fairness rules are the same for every method. All methods share the training subsample, template set, pseudo-experiment pool, architecture, and a fixed budget of tuning trials and GPU-hours. All templates come from un-oversampled data.
The key ablation separates the selection criterion from the generator. Each generator is run with its ratio chosen by AUC and again with its ratio chosen by $I_{ss}$. A second ablation separates the tail-targeted acceptance step from plain generation.
Information-matched oversampling counts as beating the baselines if it reduces yield root-mean-square error relative to the best AUC-tuned configuration. Coverage must stay within 2% of nominal. The gain must exceed the bootstrap spread across five seeds. It must hold at the 1:1000 and 1:10000 regimes.
Why not just…?
Why not just reweight the loss instead of oversampling? In expectation, random replication and class weighting optimize the same objective, so a gap between them is not expected and would not be a contribution. We include both and expect them to match. The claim does not rest on resampling versus weighting. It rests on choosing the effective ratio by downstream information rather than by AUC, and on where synthetic points are placed. If dynamic reweighting with an $I_{ss}$-selected strength matches information-matched oversampling, we will report that as the main finding: the criterion matters and the mechanism does not.
Why not just tune the ratio on validation AUC, as everyone does? This is the real threat. If AUC-selected and $I_{ss}$-selected ratios coincide, or yield nearly identical fit precision, the proposal reduces to "use oversampling," and we will say so. The paired ablation above settles this directly. The $\chi^2$ form gives a concrete reason to expect divergence at extreme imbalance: AUC weights all ranking errors equally, while yield information depends on the purest few percent of signal.
Why not just simulate more signal? In many analyses that is the correct answer, and our study offers nothing when signal simulation is cheap. The regime we target has a fixed small signal sample: an expensive simulation, or a data-driven control sample. D'Amico et al. explicitly identify this regime as a weakness of single-event likelihoods. We report every result as a function of signal training size, so a reader can see where the advantage vanishes.
Why not just apply hard cuts or a binned fit on the score? Hard cuts discard signal exposure. D'Amico et al. measured a 15–20% precision loss from cuts relative to single-event likelihoods. A binned fit on the score inherits the same information criterion, so we report it as a secondary readout to show the conclusions do not depend on binning.
Decisive Checks
The study succeeds if three things hold. Rankings of oversampling configurations by AUC must correlate poorly with rankings by toy-fit yield precision at 1:1000 and below. The quantile decomposition must show the difference is carried by the high-purity tail. The $I_{ss}$-selected ratio must track the deployment signal fraction rather than sitting at 1:1. A further sharp prediction concerns the boundary-seeking generators, the optimal-transport method and SOS: they should gain on F1 but gain little or lose on yield precision, while their tail-targeted variants should recover it.
Two outcomes would falsify the thesis. One is if all configurations converge to the same $I_{ss}$ once the network is wide enough. That would show capacity, not training distribution, is the bottleneck. The other is if the tabular quantification replication fails to show the same ordering, which would mean the effect is specific to physics data. Coverage of the toy-fit intervals is checked throughout. A precision gain that comes with undercoverage counts as a failure, not a win.
References
- Update of the search for the neutrinoless decay<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mrow><mml:mover><mml:mrow><mml:mi>τ</mml:mi></mml:mrow><mml:mrow><mml:mo>→</mml:mo></mml:mrow></mml:mover></mml:mrow></mml:mrow><mml:mi>μ</mml:mi><mml:mi>γ</mml:mi></mml:math>
- Observation of<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="italic">B</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow><mml:mo>→</mml:mo><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="italic">ω</mml:mi><mml:mi mathvariant="italic">K</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow></mml:math>and Search for Related<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi mathvariant="italic">B</mml:mi></mml:math>Decays Modes
- Signal estimation in on/off measurements including event-by-event variables
- Searching for exotic particles in high-energy physics with deep learning
- Publishing unbinned differential cross section results
- A systematic study of the class imbalance problem in convolutional neural networks
- CASIA@SMM4H'22: A Uniform Health Information Mining System for Multilingual Social Media Texts
- SMOTE: Synthetic Minority Over-sampling Technique
- Oversampling for Imbalanced Data via Optimal Transport
- SOS: Score-based Oversampling for Tabular Data
- Revive Re-weighting in Imbalanced Learning by Density Ratio Estimation
- Sampling a Longer Life: Binary versus One-class classification Revisited