Experiments that look for extremely rare decays, such as neutrinoless double-beta decay in germanium, record enormous numbers of voltage waveforms. Most come from ordinary physics events. A minority are junk: electronic discharges, mechanical vibration picked up by the electronics ("microphonics"), and pile-up. Before any physics analysis, the junk must be removed while discarding almost no real events. The current machine-learning approach for the LEGEND experiment clusters waveforms with Affinity Propagation and then trains a support-vector machine on clusters that experts have checked by hand (León et al.). It reaches 0.024 ± 0.004% physics sacrifice on high-rate data. However, it must store an N×N similarity matrix, which caps training at about 10,000 events, and it needs a human to label clusters.
We propose replacing the clustering step with a low-rank model of what good waveforms look like. Good pulses, once aligned and normalized by energy, vary along only a few directions: drift path, position, and electronics response. They therefore form a matrix that is close to low-rank. A whole anomalous waveform is a column that does not fit. The model separates the waveform matrix into a low-rank part and a few outlier columns. A penalty on the sum of singular values (the "nuclear norm," the convex stand-in for counting rank) chooses the rank automatically. An optimizer that needs only the top singular vector at each step, rather than a full singular value decomposition (SVD), lets the method stream through millions of events using memory that grows linearly with the number of events.
Core Claim
Our claim is that robust, rank-adaptive low-rank recovery can match or beat Affinity Propagation plus support-vector machine (AP-SVM) on the sacrifice/leakage trade-off for germanium data cleaning. It would do so while training on 100 to 1000 times more events and requiring no hand-labeled clusters.
The new ingredients are two. First, outlier pursuit with a column-sparse penalty, which treats each waveform as either explained or anomalous, is run with a convex–concave SVD-free solver. Second, the method is applied to streaming detector data whose background rank drifts over time. We also state the claim's main weakness up front. An anomaly class that is frequent and self-similar is itself low-rank and may be absorbed into the background. Pile-up, which makes up most of AP-SVM's 15.034 ± 0.224% high-rate leakage, is the obvious case. We test this failure mode directly rather than assume it away.
Research Program
Each triggered waveform is aligned at its rising edge and divided by its reconstructed energy. Waveforms are stacked as columns of $Y \in \mathbb{R}^{T \times N}$, with $T$ time samples and $N$ events, processed in windows of a few hundred thousand events. We solve
$$\min_{X, C} \|X\|_* + \lambda \|C\|_{2,1} \quad \text{subject to } \|Y - X - C\|_F \le \epsilon,$$
where $X$ is the low-rank model of good pulses and $C$ holds the anomalous columns. The term $\|C\|_{2,1}$, the sum of column norms, pushes entire waveforms rather than individual samples into the outlier part. The tolerance $\epsilon$ is set from the measured baseline noise. An event's anomaly score is the norm of its column of $C$, or equivalently its residual after projection onto the recovered subspace. Because that projection is an unpenalized least-squares fit, uniform singular-value shrinkage does not distort the score. We also check this empirically on pulse-shape quantities (see below).
The optimizer comes from SVD-free Convex-Concave Approaches for Nuclear Norm Regularization. That paper writes the nuclear norm through its dual, $\|X\|_* = \max_{\|U\|_2 \le 1} \operatorname{tr}(U^T X)$, and replaces the spectral-norm projection with a hinge penalty that needs only the leading singular pair. Each step then costs $O(TN)$ rather than $O(T^2 N)$. Its proximal variant handles the $\|C\|_{2,1}$ term in closed form, and its stochastic variant handles mini-batches. The original paper tested only on News20 and Facebook100 matrices. This is the first use we know of on physical sensor streams.
Between windows, the dual variable and the current subspace carry over as a warm start. This lets us track the effective rank over months of run conditions. That is exactly the setting where fixed-rank methods break: the authors of LRMC and the QR-retraction manifold method both state that they require a known rank $r$.
We also borrow a narrow idea from the Modified Nuclear Norm, which penalizes the rank of a gradient-transformed image and raised RPCA phase-transition success from 20.88% to 45.24%. We apply the nuclear norm to the time-differenced waveform, which approximates the current pulse that the A/E discriminator uses. We treat this as an ablation, not a core claim, because MNN's guarantees were proven for 2D spatial operators.
Two known risks constrain the design. The first is the sensitivity of the hinge parameter ρ reported for SECONE. We will fix ρ by a rule tied to the measured spectral norm on the first window and report sensitivity around that value. The second is power-iteration stalling when the top two singular values are nearly equal. We will log the number of inner iterations and fall back to block Lanczos if it rises.
Baselines
The do-nothing reference is the existing physics-parameter cuts: energy, A/E, and delayed charge recovery as in Arnquist et al., applied with no data-cleaning model. This shows how much any learned cleaner adds.
The strongest existing method on this exact task is AP-SVM from León et al.. It reports a maximum physics sacrifice of 0.024 ± 0.004% and maximum anomalous leakage of 15.034 ± 0.224% on high-rate detector data. On low-rate dummy-board data it reports 0.000 ± 0.021% sacrifice and 0.000 ± 0.330% leakage. We will rerun it on the same events with the same discrete-wavelet features and the same 10,000-event training cap it requires. We will also give it the same expert-labeling budget, so that the comparison isolates the clustering stage. Beating it means one of two outcomes at the same leakage level. Either sacrifice is lower at a statistically separable level, or sacrifice is equal while high-rate leakage falls clearly below 15%. The second outcome is more achievable.
The simplest serious competitor is plain truncated randomized PCA residual scoring. We fit the top $r$ principal components on the training window and score events by reconstruction residual. We will tune $r$ on the same validation split by sweeping it, giving PCA an advantage our method does not get.
The time-series state of the art in this file is RoseCDL (Yehya et al.). It is a convolutional dictionary learner with a trimmed loss and reports AUC 0.92 on TSB-UAD Apnea-ECG, against 0.47 for Matrix Profile. We run it on the concatenated waveform stream with its default trimming.
Two ablations isolate the key ingredients. One replaces the convex column-sparse model with fixed-rank factorization at the PCA-tuned rank, which tests whether rank adaptivity matters. The other drops the outlier term, which reduces to nuclear-norm denoising and tests whether robustness matters. All methods share hardware, the same tuning grid size, and the same held-out evaluation events.
Why not just…?
Why not just use truncated PCA on the waveforms? This is the real threat, and we take it seriously. If the training set is clean and conditions are stable, PCA residuals should do about as well, because the scoring step of our method is also a projection residual. Our method can only win where PCA's assumptions fail. One such case is a contaminated training set: if anomalies make up a few percent of events, they tilt the leading components. The other is a drifting rank. We will sweep injected contamination from 0% to 20% using hand-verified anomaly exemplars, and run across run periods with different conditions. If PCA with a tuned rank matches us across the whole sweep, we will conclude that the simpler tool suffices and report that. That would itself be a useful finding for LEGEND-1000 planning.
Why not just subsample Affinity Propagation, or use mini-batch k-means, to get past the memory wall? Subsampling keeps AP's 10,000-event cap, so rare anomaly classes seen only a few times in millions of events are likely missed altogether. k-means scales but needs the number of clusters fixed in advance, and it scores novelty by distance to a centroid, which is sensitive to amplitude normalization. We include mini-batch k-means distance-to-centroid as an extra scalable baseline. The rare-class recall experiment below shows whether it falls short.
Why not just use the existing A/E and delayed-charge cuts? Those cuts were built to reject specific physical backgrounds, surface alphas and multi-site events, and they do this very well: 100% alpha rejection beyond 3 mm with 0.2% bulk loss (Arnquist et al.). They were not designed for non-physical electronics artifacts. León et al. note that manual cut tuning becomes inconsistent as hardware drifts. The do-nothing baseline measures how much junk the cuts let through.
Proposed Experiments
The primary experiment uses the LEGEND Full Chain Test stand data from León et al., both high-rate detector data and low-rate dummy-board data, which requires access through the collaboration. We report sacrifice and leakage with binomial uncertainties at matched operating points, using the same definitions as the original paper.
The first decisive check is the high-rate pile-up regime. We predict that the column-sparse model will absorb some pile-up into the background, because pile-up is a superposition of good pulses. We will report this honestly and test whether a time-differenced (current-domain) variant separates it better.
The second decisive check is the contamination sweep against tuned PCA described above. It determines whether robustness and adaptive rank actually carry the method.
The third check is distortion. On calibration-source data we compare A/E and delayed-charge distributions for events the method keeps against the raw distributions. Any shift in their means beyond statistical error would show that the low-rank model is biasing pulse-shape quantities that downstream physics depends on.
The fourth check is scale. We plot wall-clock time and peak memory against N from $10^4$ to $10^7$, set against AP's quadratic memory. We also plot recall on anomaly classes that appear fewer than 50 times in the training stream.
As a public, reproducible check that does not depend on collaboration data, we run the same pipeline on TSB-UAD Apnea-ECG, cutting the signal into aligned beat windows as columns. The reference is RoseCDL's reported AUC of 0.92. We do not expect to beat it there, because ECG anomalies are not well described as outlier columns. The purpose of this run is to calibrate where the column-outlier assumption holds and where it fails.
References
- Machine learning-powered data cleaning for LEGEND: a semi-supervised approach using affinity propagation and support vector machines
- SVD-free Convex-Concave Approaches for Nuclear Norm Regularization
- Deeply Learned Robust Matrix Completion for Large-scale Low-rank Data Recovery
- Low-Rank Matrix Completion via QR-Based Retraction on Manifolds
- Beyond Low-rankness: Guaranteed Matrix Recovery via Modified Nuclear Norm
- $$\alpha $$-event characterization and rejection in point-contact HPGe detectors
- RoseCDL: Robust and Scalable Convolutional Dictionary Learning for rare-event and anomaly detection