Signal-Aware Compression of Systematic Nuisances in Beta-Spectrum Searches

Searches for new particles in nuclear beta decay look for small, sharp features in an otherwise smooth electron energy spectrum. One example is the "kink" a heavy sterile neutrino would leave where its decay branch shuts off. At high statistics these searches are limited by calibration and modeling errors, not by counting. Today experiments handle those errors in one of two ways. They write down every known error source and build a large covariance matrix, or they assume errors are smooth and filter them out by frequency, for example with wavelets. The first approach breaks when an error source was never modeled. The second breaks when the detector's energy resolution blurs the kink until it too looks smooth.

We propose a third route. We learn the low-dimensional space that systematic distortions actually occupy, then choose how many of those directions to fit using an explicit accounting of how much each one overlaps the signal. Separation then comes from direction in spectrum space rather than from frequency. Our core prediction is that this should survive energy smearing that destroys wavelet-based separation, while staying robust to error families the analyst did not anticipate. Put in two sentences: compress the space of plausible instrument errors into a few learned shapes, then fit only as many of those shapes as you can without absorbing the new-physics signal. The number to keep is chosen by measuring the overlap between each error shape and the signal template, and by testing against error types held out of training.

Core Claim

Write the binned spectrum, in units of its statistical error, as $T = T_0 + s\,K_m + B a$. Here $T_0$ is the nominal prediction, $K_m$ is the sterile-neutrino kink template for mass $m$ after detector smearing, $s$ is the mixing amplitude, $B$ is a rank-$k$ matrix of learned distortion shapes, and $a$ holds their fitted amplitudes. If $P_B$ is the projector onto the columns of $B$, then the information about $s$ that survives fitting the nuisances, and the bias a residual unmodeled distortion $\delta$ induces, are

$$ I_s = K_m^\top (I - P_B) K_m, \qquad b_s = \frac{K_m^\top (I - P_B)\,\delta}{K_m^\top (I - P_B) K_m}. $$

These two quantities state the trade-off precisely. A larger $k$ shrinks $b_s$ but can rotate $B$ toward $K_m$ and erase $I_s$, which is the "over-profiling" failure. A smaller $k$ keeps $I_s$ but lets $\delta$ leak into $s$ as a false signal.

The claim is that this trade-off can be optimized per mass hypothesis. The $k$ that minimizes the worst-case held-out bias divided by $\sqrt{1/I_s}$ is a better operating point than any of the following:

  • a fixed polynomial shape factor;
  • a full analytic covariance matrix, which is equivalent to profiling all modeled directions with Gaussian priors;
  • a frequency cutoff.

We expect the advantage to grow with energy resolution width, because smearing pushes $K_m$ toward low frequency. Frequency separation then fails, while directional separation need not, provided the learned distortion subspace and the smeared kink are not aligned.

This sharpens an observation from collider anomaly detection. Anomaly-preserving contrastive neural embeddings found that latent dimensionality that is good for modeling background (64 dimensions) is bad for discovery (4 dimensions are optimal). That paper chose dimension empirically. Here the corresponding choice has a closed-form diagnostic, $I_s$, and an adversarial validation protocol.

Research Program

Building the ensemble of distortions. The first component is a library of physically motivated spectral perturbations for a tritium spectrometer. We generate them by varying the error sources KATRIN analyses model explicitly:

  • tritium activity and column density drift;
  • energy-loss functions in the source gas;
  • magnetic field and high-voltage fluctuations;
  • energy-dependent detector efficiency;
  • theoretical shape corrections.

These follow Search for keV-scale sterile neutrinos with the first KATRIN data and Improved eV-scale sterile-neutrino constraints from the second KATRIN measurement campaign. To stand in for errors no one modeled, we add a generic family: smooth random distortions with a tunable correlation length.

Learning the subspace. Each perturbation is whitened by the statistical error. A singular value decomposition of the stacked perturbations yields the ordered basis $B$. Distortions are grouped by physical origin so that whole families can be withheld.

Choosing the rank. For each candidate mass $m$ and each smearing width, we compute $I_s(k)$ and the held-out bias $b_s(k)$. The bias is measured in leave-one-family-out fashion: the basis is built without, say, the energy-loss family, then tested on it. We select $k^*(m)$ by minimizing the larger of the statistical error and the worst-case held-out bias. The principal angles between $K_m$ and the span of $B$ serve as a diagnostic. Masses where those angles collapse are flagged as fundamentally degenerate with systematics, rather than being reported as limits.

Fitting. The final fit profiles only $k^*$ amplitudes. That leaves a handful of parameters instead of a dense covariance over hundreds of bins, which also targets the strong correlation between active and sterile mass-squared that KNM2 had to resolve with grid scans.

Relation to parameter-space reduction. The quantum-circuit literature supplies the lesson, not the machinery. Symmetric Pruning in Quantum Neural Networks showed that cutting a search space to its relevant subspace changed the parameter count needed to reach the well-behaved optimization regime from about 800 to about 50 on the transverse-field Ising model. Exploiting symmetry in variational quantum machine learning showed the price: a restricted model cannot represent components outside its subspace. Our nuisance basis is the analogous restriction, applied to the error model. The kink is the out-of-subspace component we want the restriction not to represent.

We deliberately do not use quantum circuits. Both of those papers test idealized spin systems in noise-free simulation, and nothing in the file indicates a quantum advantage for one-dimensional spectral fits.

Baselines

Unprotected global fit. The naive reference is the standard global least-squares fit with a polynomial shape factor in $(Q-E)$. We give it the same simulated spectra and fit window, and its polynomial order is tuned on the same held-out families we use to pick $k^*$. Beating it means a lower false-positive rate at matched sensitivity. Wavelet approach to search for sterile neutrinos in tritium β-decay spectra reports that this approach is substantially biased by a 10% polynomial distortion.

Wavelet method. The strongest model-independent competitor is the Daubechies-18 wavelet approach from that same paper. It reports a statistical sensitivity of $\sin^2\theta \approx 10^{-6}$ (90% CL) for 4–15 keV masses, assuming $10^{11}$ decays/s over three years. It reports limits essentially unchanged under a 10% polynomial distortion. It also requires about 100 eV resolution, with the signal vanishing above 500 eV. We reimplement it on identical simulated spectra.

  • At 100 eV resolution, matching its sensitivity while meeting coverage on held-out families counts as parity.
  • At 300–1000 eV, retaining a nonzero, correctly covered sensitivity where its wavelet power vanishes counts as beating it.

Full analytic covariance. The strongest model-based competitor is full covariance-matrix profiling as used in the KATRIN analyses. Those report $\sin^2\theta < 5\times10^{-4}$ (95% CL) at 0.3 keV on First Tritium data, and sensitivity to $m_4^2 \lesssim 1600$ eV² and $|U_{e4}|^2 \gtrsim 6\times10^{-3}$ on KNM2. We build the covariance from the training families only, so it receives exactly the same information as our learned basis. Beating it means better coverage and lower bias on the withheld families, at no more than a modest sensitivity cost.

Ablations.

  • Signal-blind PCA: fix $k$ by explained variance alone, ignoring $I_s$. This isolates the signal-aware rank choice.
  • No generic family: drop the generic smooth distortions. This isolates the contribution of the "unknown systematic" family.

Why not just…?

Why not just use the full covariance matrix? If every systematic is modeled correctly and is Gaussian, covariance profiling is optimal, and our method can at best match it. This is a real threat and we state it plainly. Our gain exists only under misspecification. That is exactly the regime the leave-one-family-out test probes, and it is the one the file identifies as unresolved: correlated systematics escaping the parameterization, and over-profiling. If covariance coverage holds on withheld families, we will report that the compression buys only computational savings.

Why not just use wavelets? Wavelets assume systematics live at large scales and kinks at small ones. The wavelet paper itself shows this assumption collapsing beyond 500 eV resolution, which is the regime of wide-window measurements. The resolution scan directly tests whether directional separation survives where frequency separation does not. If the smeared kink becomes nearly parallel to the learned subspace, the principal-angle diagnostic will show this, and the method will refuse to quote a limit there instead of claiming one.

Why not just use a contrastive embedding with a goodness-of-fit test, as at the LHC? That pipeline earns its keep on 57-dimensional event kinematics, where raw testing gave 0σ. A binned one-dimensional spectrum already has an explicit forward model and a linear signal template, so a learned nonlinear embedding adds training variance without obvious benefit. We include it as an optional comparison only if the linear subspace fails.

Proposed Experiments

Main test. On the simulated keV-scale spectrum (10^11 decays/s, three years, 4–15 keV masses), we scan resolution from 100 to 1000 eV. For each method we report:

  • the expected 90% CL sensitivity curve;
  • the false-positive rate and the coverage of nominal intervals under each withheld systematic family, injected at amplitudes up to the 10% level used by the wavelet paper.

Second test. We repeat the analysis on simulated integral spectra matching the First Tritium configuration (1.6 keV window) and the KNM2 configuration (40 eV window). The aim is to check that, with all families known, our method approximately reproduces the published sensitivities. Real KATRIN data are not assumed to be available.

Decisive check. The proposal is wrong if either of two things happens:

  • at resolutions above 500 eV, signal-aware compression yields either no usable sensitivity or undercoverage on held-out families;
  • signal-blind PCA does as well, showing that the $I_s$-based rank choice adds nothing.