Physics experiments that hunt for very rare events, such as neutrinoless double-beta decay or a new particle buried under ordinary background, increasingly rely on neural classifiers to separate signal from background. Those classifiers are trained on simulation, because no real signal event can ever be labeled. When the simulator gets the detector slightly wrong, the classifier's output on real data drifts. That drift is then absorbed as a systematic uncertainty in the final likelihood fit, and it erodes exactly the sensitivity the network was meant to buy. Meanwhile the experiment holds large amounts of real but unlabeled detector data: background-only sidebands outside the signal region, and calibration runs with radioactive sources whose event topologies are known. Today these mostly sit unused during training.
We propose to train the classifier with a teacher–student scheme. A "teacher" network assigns soft labels to a large pool of simulated and real unlabeled events. A "student" network learns from those labels. The teacher is then corrected by how much the updated student improves the expected discovery significance of the downstream statistical test, measured only on held-out real control data. The task score that judges the labels is the physics figure of merit itself, not accuracy. The judging data is real detector data from regions where the answer is known, not simulation. We also add an explicit penalty whenever the student manufactures a fake signal in background-only data. That penalty addresses the main reason physicists distrust learning on real data.
Core Claim
Bi-level pseudo-labeling, introduced as Meta Pseudo Labels, works as a two-step loop. The student takes a trial gradient step on the teacher's labels. The student's loss on a clean labeled set is then differentiated back through that step to update the teacher. Existing versions judge that step with cross-entropy or squared error on a clean labeled split. This holds for the original image work, the first-order variant in Semi-Supervised Learning with Meta-Gradient, and the scientific-surrogate version Uncertainty-Informed Meta Pseudo Labeling. Rare-event searches break both assumptions:
- No clean labeled real signal exists.
- At signal-to-background ratios of $10^{-4}$ or below, a mixture cross-entropy is dominated by background, so the teacher gets almost no signal-related feedback.
Our claim is that both problems are solved by one design change. The outer objective becomes a differentiable approximation of the profile-likelihood discovery significance. The background score distribution in that significance is estimated from real sideband data. The signal efficiency is estimated from simulated signal checked against real calibration proxies. Because significance depends on the signal and background counts in each score bin jointly, the rare class cannot be diluted away the way it is in a summed loss.
Concretely, let $s_k(\theta')$ and $b_k(\theta')$ be the expected signal and background counts in score bin $k$ after the virtual student update $\theta'$. The bins are made differentiable by soft assignment. The outer loss is
$$ L_{\text{outer}} = -Z_A(\theta') + \lambda\,\frac{|\hat N_S^{\text{ctrl}}(\theta')|}{\sigma_{N_S}^{\text{ctrl}}}, \qquad Z_A^2 = 2\sum_k\Big[(s_k+b_k)\ln\big(1+s_k/b_k\big)-s_k\Big]. $$
Here $Z_A$ is the median discovery significance obtained by evaluating the likelihood-ratio statistic on its expected ("Asimov") dataset. The second term is the fitted signal yield $\hat N_S^{\text{ctrl}}$ in a held-out, background-only real control region, divided by its uncertainty $\sigma_{N_S}^{\text{ctrl}}$, and $\lambda$ trades the two terms off. The final analysis still uses the experiment's unbinned extended likelihood, with the classifier score entering as one more observable. The binned form appears only inside training, where it must be differentiable.
Research Program
Data partition. The work starts with a strict partition of real data. Partition A contains sideband and calibration events that the student trains on through teacher labels. Partition B is a disjoint real control set that serves only as the outer judge. Partition C is a final held-out region; in the experiments it receives injected signal. The signal region proper stays blinded throughout. This split is what keeps the student from fitting its own judge, and it mirrors how experiments already use orthogonal control regions, as the MAJORANA Project and unbinned signal-extraction work such as Unbinned Maximum Likelihood Estimation for Signal Extraction do.
Teacher and meta-gradient. The teacher outputs soft signal-probabilities on simulated events and on real Partition-A events. To avoid second-order cost, the meta-gradient uses the first-order Taylor approximation of Xiao et al.. We will also check on small models that this matches the full second-order gradient.
Signal side of the objective. Signal expectations $s_k$ come from simulated signal reweighted by a small set of correction factors. These factors are fit to the calibration proxy, for example single-site-like events from a source peak in a xenon detector of the EXO-200 type. We do not trust the proxy as a stand-in for signal shape; it constrains only efficiency.
Analysis questions. We will study three things:
- Whether the significance-based outer loss avoids the degenerate "label everything background" solution that the class-imbalance analysis predicts for cross-entropy.
- How $\lambda$ trades sensitivity against spurious signal.
- Whether the teacher's corrections concentrate on the simulator's known weak spots. In a controlled testbed these are known by construction.
Baselines
Do nothing. The naive reference is a supervised deep network trained only on simulation, in the style of Baldi et al.. Its score enters the final likelihood fit, and the mismatch is absorbed by floating nuisance parameters. This is current practice; EXO-200 reports roughly a 25% sensitivity gain from its simulation-trained network (EXO-200). Beating it means a higher median significance (or a tighter median limit) on Partition C, at equal or better coverage of injected-signal tests.
Adversarial domain adaptation. The strongest existing simulator-to-data method in the file is the gradient-reversal network of Clavijo et al.. It trains features that a second network cannot use to tell the two event generators apart. It reports target-domain AUC rising from 0.657 to 0.760, and relative signal-purity uncertainty falling from 47% to 0.3%. We will run it on the same partitions, with the same architecture, the same hyperparameter search budget, and its own KS-distance criterion for choosing the adversarial weight. On their generator pair, beating it means exceeding 0.760 target AUC and, more importantly, giving higher significance at matched spurious-signal rate.
Closest method. The closest method is standard Meta Pseudo Labels with a cross-entropy outer loss on the calibration proxy plus labeled sidebands. It is identical to our method except for the outer objective, and is run with identical compute. No reported result on Lacuna was found for this method on any physics benchmark.
Key ablation. The ablation that isolates the bi-level ingredient is single-level fine-tuning. Starting from the simulation-trained network, we directly maximize the same $-Z_A + \lambda\,|\hat N_S|/\sigma$ objective, computed on Partitions A and B merged, with no teacher.
Secondary ablations. We also remove the spurious-signal penalty and replace soft binning with hard binning plus straight-through gradients.
Why not just…?
Why not just pseudo-label real data with a confidence threshold and retrain? Thresholding cannot detect a systematic shift. A classifier that is confidently wrong on mismodeled background tails reinforces that error, which is the confirmation-bias failure that Meta Pseudo Labels illustrates on toy data. UMPL reports the same collapse in physics surrogates. In a rare-event search this failure shows up as spurious signal. We include static self-training as a baseline and expect its spurious-signal rate on background-only injections to be its decisive weakness.
Why not just use adversarial domain adaptation? It is a single-level method with no feedback from the downstream test. Clavijo et al. note that it struggles under label shift, which is precisely the rare-signal situation: the real data's signal fraction is unknown and tiny. Under these conditions the domain critic can penalize genuine signal features. This alternative is a real threat on pure covariate shift. Our testbed therefore includes one pure covariate-shift condition and one shift that is correlated with the signal, and we report both honestly.
Why not just fine-tune directly on the significance loss with the real control data? This is the most serious alternative. If it matches our method, the teacher is unnecessary and we will say so. Our reason to expect a gap is that the real control data is small and biased toward proxy topologies. Direct fine-tuning lets the network fit those few events. The bi-level loop instead uses them only as a judge, while the teacher spreads corrections across the large simulated pool. The ablation above settles this, comparing gaps between Partition B and Partition C and comparing spurious-signal rates.
Why not just leave mismodeling to nuisance parameters in the fit? That is the do-nothing baseline. Its cost is measurable as the widening of the profile likelihood when nuisances float. We report significance both with nuisances fixed and with them floated, so the recovered sensitivity is explicit.
Proposed Experiments
Testbed 1: controlled HIGGS shift. The primary testbed is the public HIGGS benchmark from Baldi et al., which reports 4.9σ for a deep network versus 2.5σ for a shallow one on low-level features. We turn it into a controlled simulator-versus-data problem. The "real" detector gets known distortions: a jet energy-scale shift, extra smearing, and a background-tail component absent from the "simulation." Signal is injected at prevalences from $10^{-3}$ to $10^{-5}$. A sideband is defined in reconstructed mass, and a small "calibration" set is labeled only in a proxy region.
Testbed 2: generator pair. The second testbed is the generator pair of Clavijo et al., MadGraph versus Powheg for ttH with H→bb̄. We use their samples if available and otherwise regenerate with the same tools; this provides a realistic shift that we did not design ourselves.
Metrics. We measure median expected significance and median 90% CL upper limit on Partition C. We also measure coverage and bias of the fitted signal yield over several hundred pseudo-experiments, at both zero and nonzero injected signal. The spurious-signal rate is the fraction of background-only pseudo-experiments exceeding 3σ. Target AUC is recorded for comparison with Clavijo et al.
Decisive checks. The method succeeds only if all of the following hold:
- It beats the do-nothing baseline and adversarial adaptation in median significance at matched spurious-signal rate.
- It beats single-level significance fine-tuning on Partition C, not just on Partition B.
- It keeps fitted-yield coverage within statistical error of nominal.
- The cross-entropy-judged Meta Pseudo Labels variant degrades sharply as prevalence falls toward $10^{-5}$, while ours does not. This confirms that the outer objective, not the bi-level structure alone, removes the background dominance.
A positive result on the HIGGS and generator-pair testbeds would justify a collaboration-internal test on real calibration and sideband data from a running low-background experiment.
References
- Meta Pseudo Labels
- Semi-Supervised Learning with Meta-Gradient
- Uncertainty-Informed Meta Pseudo Labeling for Surrogate Modeling with Limited Labeled Data
- The MAJORANA Project
- Dalitz plot analysis of<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msubsup><mml:mi>D</mml:mi><mml:mi>s</mml:mi><mml:mo>+</mml:mo></mml:msubsup><mml:mo>→</mml:mo><mml:msup><mml:mi>K</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>K</mml:mi><mml:mo>−</mml:mo></mml:msup><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup></mml:math>
- Search for Neutrinoless Double-<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>β</mml:mi></mml:mrow></mml:math>Decay with the Complete EXO-200 Dataset
- Searching for exotic particles in high-energy physics with deep learning
- Adversarial domain adaptation to reduce sample bias of a high energy physics event classifier