Separating New Physics from Detector Drift with Block-Term Tensor Models

Precision experiments now collect so much data that random counting noise is no longer the limit. The limit is the residual systematic: a small, unmodeled mismatch between the real detector and the simulation used to interpret it. One example is a calorimeter region whose energy scale drifts over months. When such a mismatch lands in the spectrum where a search looks for a bump, it can fake a discovery or hide a real one. The standard remedy is to list every known systematic effect as a nuisance parameter, meaning a knob in the fit with a simulated "template" for how it distorts the spectrum. That remedy fails exactly when the systematic is one nobody thought to list.

This note proposes to use a physical asymmetry instead of a longer list. A genuine physics signal does not know how the detector is segmented or what week it is. Its spectral shape is the same in every run period, and its yield scales with recorded luminosity. Detector systematics, by contrast, vary across segments and drift in time. If the binned data are arranged as a three-way array (observable bin × detector segment × run period), a signal and a drift occupy algebraically different structures. A block term decomposition can then separate them without knowing the drift's shape in advance. Block term decomposition writes a multi-way array as a sum of components, each with a little internal rank rather than a single outer product of vectors. The same decomposition also says, through known uniqueness theorems, when the separation is mathematically possible at all.

Core Claim

We model the residual systematics in a segmented, multi-period measurement as a small, unknown number of rank-$(1, L_r, L_r)$ blocks. Each block has one spectral distortion shape, and its amplitude across segments and periods forms a low-rank matrix. The signal is a separate, time-invariant term whose time profile is fixed to the luminosity. Concretely, for counts $X_{ijk}$ in observable bin $i$, segment $j$ and period $k$,

$$X_{ijk} \approx \mu^0_{ijk} + \sigma\, s_i\, g_j\, \ell_k + \sum_{r=1}^{R} a_{ir}\, (B_r C_r^{\top})_{jk}.$$

Here $\mu^0$ is the nominal simulated expectation including known nuisances. $\sigma$ is the signal strength, $s$ is the signal's spectral shape, $g$ is its unknown per-segment acceptance, and $\ell$ is the known per-period luminosity. $a_r$ is the spectral template of residual systematic $r$. $B_r$ (segments × $L_r$) and $C_r$ (periods × $L_r$) give its low-rank spatiotemporal pattern. We constrain $C_r^{\top}\ell = 0$, so the residual blocks carry only time-varying structure.

That constraint states the method's limit honestly. A static miscalibration with exactly the signal's shape is indistinguishable from signal by any data-driven method, so it stays with conventional nuisance profiling. A drifting systematic, by contrast, becomes separable. The rank-$(1, L, L)$ form matches the three-way identifiability theory of On identifiability of 3-tensors of multilinear rank (1, L_r, L_r). That paper derives exact and generic uniqueness conditions for sums of such terms on $I \times J \times K$ arrays, including a general $R$-term generic result (its Theorem 1.9). We deliberately stay in three modes so that we sit inside proven theory rather than in the open four-way-and-higher case.

What is new is not "apply tensors to physics." It is two things. First, a detection method in which the shape of the unknown systematic is learned from its spatiotemporal inconsistency. Second, a design rule that follows from the theorems: it tells an analysis how many segments and periods are needed before an unknown drift of a given complexity can no longer mimic a signal.

Relation to Prior Work

Three groups of prior work shape this proposal, in different roles.

The first group motivates the gap. In the research file, the systematics-aware machine-learning tools for physics are all attached to a single learned score. Robust Anomaly Detection for Particle Physics Using Multi-Background Representation Learning uses adversarial decorrelation (NuRD) to stop an anomaly score from sculpting the jet-mass distribution, and it still leaves a Jensen-Shannon divergence of 0.118. Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference restores coverage when nuisance distributions shift, but only for nuisances the analyst has named. Simulation-Based Inference Methods for Particle Physics identifies simulator misspecification as the open weakness of likelihood-free inference. None of these methods uses the segment and run-period structure of the data to detect systematics nobody listed, which is what we target.

The second group supplies the method. Block-Term Tensor Decomposition: Model Selection and Computation provides BTD-HIRLS. Its two-level group-sparsity penalty prunes whole blocks and individual columns, recovering block counts and ranks with over 90% success on synthetic data at 5–20 dB. We reuse it to choose $R$ and $L_r$, replacing its squared-error loss with a Poisson deviance, which iteratively reweighted least squares handles naturally. A variable projection method for block term decomposition of higher-order tensors reports roughly twice the convergence speed of alternating least squares, and we use it for the many refits that toy calibration requires. Common and Discriminative Subspace Kernel-Based Multiblock Tensor Partial Least Squares Regression splits multi-camera data into shared and camera-specific subspaces. Our signal-versus-drift split is the physics analogue, with "shared" given a precise meaning: time-invariant and proportional to luminosity.

The third group constrains the design. Higher-Order Block Term Decomposition for Spatially Folded fMRI Data shows that flattening a multi-way array and fitting rank-1 components produces "phantom" sources when true components have internal rank, while BTD is robust to overestimating that rank. This is why we neither flatten nor use rank-1 blocks. Bhaskara et al. warns that even when uniqueness holds, practical algorithms may fail near the theoretical threshold. That is why we measure empirical recovery against the theoretical boundary rather than assuming the two coincide.

Research Program

The work has three parts.

The first part is the estimator. We will build a Poisson-likelihood block term fit with the luminosity-locked signal term, the time-orthogonality constraint on residual blocks, and BTD-HIRLS-style hierarchical sparsity. Smoothness in the period mode is an optional penalty, which we will ablate. The test statistic for $\sigma$ is a profile likelihood ratio. Because the fit is non-convex, we will calibrate it with pseudo-experiments (toy datasets drawn from the fitted null) rather than with asymptotic formulas.

The second part is the design rule. Using the generic identifiability conditions in Cheng et al., we will map, for given numbers of bins $I$, segments $J$ and periods $K$, the largest total residual complexity $\sum_r L_r$ for which signal and residual blocks are generically unique. We will then test whether the algorithm actually achieves recovery up to that boundary, and quantify the gap that Bhaskara et al. warns about.

The third part is the benchmark, which we will release. It starts from the LHC Olympics 2020 R&D sample described in The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics: one million QCD dijet events plus a $Z' \to XY$ signal at 3.5 TeV. We will assign background events to pseudo-run periods with a chosen luminosity profile. We will define segments by jet pseudorapidity region, the angular position along the beam axis, which is a natural proxy for detector region. We will then apply jet-energy-scale and resolution drifts that vary by segment and period, with shapes and patterns hidden from the analyst. The observable is dijet invariant mass, so a drifting scale can create a false bump.

The Black Box 2 sample, a background-only blinded set, serves as an additional null. We will inject the same drifts into it to measure false-discovery behavior on data the method was never tuned on.

Baselines

The naive reference is a bump hunt on the summed mass spectrum, fit with the nominal expectation and only the known, static nuisance parameters. This is what a drift-blind analysis does. Its false-discovery rate under injected drift is the number every other method must cut.

The strongest conventional competitor is a profile-likelihood fit that gives every segment–period cell its own energy-scale and resolution nuisance, using the true simulated templates. This baseline has more knowledge than our method, since it is told the template shapes. In the regime where the injected drifts match those templates, it should win or tie. The fair comparison runs both methods on the same binned arrays, with the same pseudo-experiment budget for calibration and the same toy count. We then add drifts whose spectral shape is not in the template list. We beat this baseline if, under unlisted drifts, we hold the false-discovery rate at nominal while keeping signal-strength bias and median expected significance within a small margin of its oracle-template performance.

The ablation baselines isolate the key ingredient. The first is truncated singular value decomposition of the flattened residual matrix (bins × segment–periods), removing the leading modes. The second is a rank-1 decomposition (canonical polyadic) of the same three-way array. The third is our model with the luminosity lock and time-orthogonality removed. The last ablation tests whether the time-structure prior, rather than the tensor form alone, does the work. Each ablation gets the same rank-selection budget, so that each can choose its number of components by the same cross-validated deviance criterion.

Learned-score anomaly detectors are a further comparison point, specifically FORCE from Anomaly Detection in Collider Physics via Factorized Observables and the contrastive pipeline of Anomaly-preserving contrastive neural embeddings for end-to-end model-independent searches at the LHC. They are complementary rather than direct competitors. We will run FORCE's selection upstream and ask whether our drift model preserves its sensitivity under drift.

The file reports FORCE raising local significance from roughly 1–2σ to over 5σ at a 0.5% signal fraction, and the supervised SimCLR Transformer reaching 12.1σ on a black box. Both results are on drift-free data. No reported result was found on Lacuna for any method on LHC Olympics data with injected segment- and time-dependent detector drift. Our benchmark therefore defines this number for the first time, and the drift-free figures above are upper references, not targets.

Why not just…?

Why not just add per-cell nuisance parameters, as experiments already do? When the drift's spectral shape is known, this is the right answer, and our oracle-template baseline is exactly this approach. The threat is real: if the analyst's template list happens to cover everything, our method only adds variance. The experiment settles it by sweeping how far the injected drift shapes depart from the listed templates. Our claim is limited to the departure regime. We will report honestly the point at which the two methods cross over.

Why not just flatten and remove leading singular vectors of the residuals? This is the simplest data-driven competitor, and it can absorb drifts. However, a matrix factorization of bins × (segment–period) cannot distinguish a time-invariant spectral mode from a drifting one. In the presence of strong drift, a leading mode can therefore absorb part of the signal itself. Chatzichristos et al. also found that flattening loses multi-way structure and causes cross-talk between components. The truncated-SVD ablation measures signal absorption directly: we will compare the recovered $\sigma$ with the injected value.

Why not just use an adversarially decorrelated or mass-conditional anomaly score? These methods attack a different systematic, namely a selection that sculpts the background. They do not address a detector that drifts after selection. NuRD's residual JSD of 0.118 also shows that adversarial decorrelation is incomplete even for its own target. Conditioning a threshold on mass would fix sculpting, but it does nothing about a scale shift that moves events between mass bins in some periods only. Our benchmark's drifts are applied after any selection, so this distinction is tested rather than asserted.

Why not just use conformal or nuisance-aware prediction sets? NAPS, from Masserano et al., keeps 90% coverage where standard sets fall to 40–60%. However, it conditions on nuisances that are specified in advance. It could be stacked on top of our fit once our method has discovered the residual blocks, but it cannot discover them.

Proposed Experiments

The first experiment checks the theory. On synthetic $I \times J \times K$ arrays, we will vary $\sum_r L_r$ across the generic identifiability boundary from Cheng et al. and measure three things: recovery of the residual templates (correlation with the true $a_r$), signal-strength bias, and rank-selection accuracy. This should be compared against the over-90% rank recovery that BTD-HIRLS reports for squared-error data. The decisive outcome is a sharp loss of recovery near the predicted boundary. If recovery fails well below it, the design rule is unusable in practice.

The second experiment is the main LHC Olympics benchmark. We will vary the signal fraction, the drift amplitude, and how far the drift shapes depart from the listed templates. For each method we will report the false-discovery rate on background-only and Black Box 2 pseudo-experiments, the bias and interval coverage of $\sigma$, and median significance at fixed false-discovery rate.

The single most decisive check is this: with an unlisted drift strong enough to make the naive and per-cell-template fits report spurious local significance, does our method return nominal false-discovery rates while recovering an injected $Z'$ at the expected significance? And does the no-luminosity-lock ablation fail that same test? If the ablation passes, then the tensor form alone is not the reason the method works, and the proposal's central claim about time structure is wrong.

A final sanity run removes all drift. Our method should then match the nominal fit within pseudo-experiment error, showing that the extra flexibility costs little when nothing is wrong.