Subtracting Chance: Null-Corrected Linear Attention for Forecasting Thousands of Series

When a forecasting model must decide which of hundreds of sensors helps predict a given sensor, it judges similarity from a short recent window, often 96 time steps. With few time points and many candidate partners, some unrelated series will look strongly similar purely by chance. This is the high-dimensional spurious-correlation problem from statistics, here appearing inside a neural network. Linear attention is a cheap variant of attention whose cost grows linearly, not quadratically, with the number of items compared. It is the natural tool for scaling such models to thousands of series, but it is also the variant least able to ignore chance similarity, because it behaves like a weighted average over every series.

This note proposes a specific test of that weakness and a cheap correction for it. The working hypothesis is that linear attention fails with many variables mainly through bias: it gives a positive baseline weight to every series, and that weight is inflated for series that share smoothness or periodicity with the query even when they are unrelated. Softmax attention instead fails mainly through variance: it locks onto the single largest chance similarity. If the hypothesis holds, the bias can be estimated and removed inside the layer at linear cost. The method is to compute what attention would output if every pairing between series were desynchronized, which gives a "chance-level" memory, and to subtract it.

Core Claim

In a variate Transformer, each whole input series becomes one token, and attention runs across series. The best-known example is iTransformer, which represents each variable's lookback history as a token and lets attention model dependencies between variables. With linear attention, the cross-series computation compresses into a small memory matrix $S = \sum_{j=1}^{V} \phi(k_j) v_j^\top$. Here $V$ is the number of series, $k_j$ and $v_j$ are the key and value vectors of series $j$, and $\phi$ is the positive feature map that replaces the softmax. The output for series $i$ is $\phi(q_i) S$ divided by a normalizer, where $q_i$ is the query vector of series $i$.

We construct a surrogate memory $S_{\text{null}}$. It is built from keys computed on randomly circularly shifted copies of each input series, averaged over $K$ random shifts, and paired with the original values. A circular shift keeps each series' own autocorrelation and spectrum but destroys its time alignment with other series. The match between $q_i$ and a shifted key therefore measures similarity that would exist even if the two series were unrelated. The corrected layer is

$$ y_i = \frac{\phi(q_i)\,\big(S - \lambda\, S_{\text{null}}\big)}{\phi(q_i)^\top z}, $$

where $z = \sum_j \phi(k_j)$ is the usual normalizer and $\lambda \in [0,1]$ is a learned scalar per attention head. Setting $\lambda = 0$ recovers ordinary linear attention. The cost is $K+1$ key passes, so the layer remains linear in $V$.

The novelty lies in where the correction acts. Mean-centering keys removes one global offset. The surrogate removes a pair-specific chance offset that depends on whether two series share marginal structure such as a daily cycle or slow drift. In statistics this is the main source of "nonsense" correlation between autocorrelated series.

How this sits relative to prior work

Three papers motivate the proposal. The first is Robust Inter-Series Dependency Modeling for Time Series Forecasting via Information-Theoretic Alignment, which introduces CGTFra. That work diagnoses spurious cross-channel correlations in variate Transformers and counters them with a learned dynamic graph plus an information-bottleneck alignment loss. The authors state that in very high-dimensional settings shallow attention still puts weight on coincidental noise that their loss struggles to filter, and their method stays quadratic in $V$. We keep linear cost and act on chance alignment directly, not through a learned graph.

The second is Why Softmax Attention Outperforms Linear Attention. It proves that linear attention averages tokens and blurs an isolated spike into background noise, which is the averaging bias we target. The third is High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification. It shows that softmax attention detects weak sparse signals far more efficiently than pooling, but can fall into an "alignment trap" where the query locks onto noise. That trap is the variance failure we predict for softmax as $V$ grows. Neither paper studies linear attention with many correlated, autocorrelated tokens, and that regime is what we characterize.

One paper constrains the design. Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality introduces ELinFormer, which makes linear attention across up to 883 traffic sensors competitive by matching the entropy of softmax attention. Matching entropy sharpens the weights but does not ask whether sharp weights land on genuinely related series, so ELinFormer is the direct linear-attention competitor. Simple linear attention language models balance the recall-throughput tradeoff introduces BASED and shows that linear attention needs help, in the form of exact local windows, to retrieve sparse information. This supports our premise that plain linear attention is too diffuse.

The statistics literature supplies the method. The surrogate is an in-layer, learned analogue of the Goodness of Spurious Fit idea from Guarding against Spurious Discoveries in High Dimensions. That paper calibrates a fit against how well covariates could fit the response under independence, which is the same move as comparing real attention to attention under desynchronization. Variance Estimation Using Refitted Cross-validation in Ultrahigh Dimensional Regression shows that selecting and fitting on the same sample absorbs realized noise. Its naive estimator has bias −0.488 against −0.017 for refitted cross-validation at $n=50, p=1000$. This is our reason to expect that within-window similarity overstates dependence. A Selective Overview of Variable Selection in High Dimensional Feature Space documents that maximum chance correlations rise with dimension.

The delta rule studied in Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions lets a recurrent memory overwrite stale content over time. Our tokens are unordered series, so that mechanism does not apply directly.

Research Program

The first component is a controlled synthetic study designed to separate bias from variance. We generate $V$ series as independent autoregressive processes with heterogeneous spectra, some sharing a common period. Each target series is given $s$ true partners whose lagged values drive it. We sweep $V$ from 50 to 2000 and the lookback length $L$ from 48 to 720.

Two quantities are of interest. If an unrelated pair has roughly Gaussian sample correlation with spread $1/\sqrt{L}$, the largest chance correlation among $V$ candidates grows like $\sqrt{2\log V / L}$, which is the standard extreme-value result for Gaussian maxima. That is the scale at which softmax should fall into the alignment trap. For linear attention, the true partners receive a share of attention mass of roughly $s/V$ multiplied by their excess similarity. The prediction is that linear attention's error is controlled by $V$ and by shared spectral structure, while softmax's error is controlled by $\log V / L$.

We measure attention mass on true partners, attention mass on spectrally matched non-partners, and forecast error relative to an oracle that sees only the true partners. The first deliverable is a phase diagram over $(V, L)$ showing which failure dominates for each attention type.

The second component is the null-corrected layer described above, dropped into an iTransformer backbone in place of its attention. We vary three design choices to see which matters: the number of surrogate shifts $K$ (1, 4, 16), whether shifts are drawn at random or forbidden to be near multiples of the dominant period, and whether $\lambda$ is learned or fixed.

The third component is a real-data stress test that makes spuriousness measurable. To each benchmark we append $M$ distractor series, which are phase-randomized copies of real series. They have the same spectrum as a real series but no time alignment with anything, so any attention they receive is by construction spurious. We sweep $M$ up to several times $V$ and record how forecast error and attention mass on distractors change for each method.

Baselines

The naive reference points come first. One is a channel-independent model that forecasts each series from its own history alone, with no cross-series mixing; we use a per-series linear forecaster of the DLinear type. If spurious cross-series dependence is the whole story, this baseline wins. The other is iTransformer with plain linear attention, which is exactly our layer with $\lambda=0$.

The strongest linear-cost competitor is ELinFormer from Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality. We run its entropy-aware attention inside the same iTransformer backbone, so the attention layer is the only thing that differs. The file reports for it only a "36.7% reduction in memory usage" and an "18.8% reduction in training time" versus STAEformer on PEMS03. No absolute MAE or MSE value was found on Lacuna for ELinFormer, so we reproduce it.

The strongest method aimed at spurious cross-series dependence is CGTFra from Robust Inter-Series Dependency Modeling for Time Series Forecasting via Information-Theoretic Alignment. It reports an MSE reduction of 5.1% over DUET on Traffic and MSE reductions of up to 7.4% on Weather and ECL when attached to iTransformer. We also include full softmax iTransformer as the quadratic accuracy reference. As a further strong reference, we use ReFocus from ReFocus, which reports MSE reductions over iTransformer of 4% on Traffic and 6% on ECL. No absolute MSE for iTransformer on these datasets appears in the file, so every method is retrained under one protocol.

That protocol fixes lookback 96, horizons 96 to 720, the same splits, the same hyperparameter search budget per method (an equal number of trials over learning rate, width and depth), and five seeds.

Two ablations isolate the key ingredient. The first is mean-centered keys: subtract $\frac{1}{V}\sum_j \phi(k_j)$ from every key. This removes the global averaging offset at the same cost, so it separates "remove uniform bias" from "remove pair-specific chance alignment". The second is a mismatched null, in which values are shuffled across series instead of keys being shifted in time. It tests whether gains come from the specific desynchronization null or from any subtraction.

Beating the baselines means three things. Our layer must match or beat ELinFormer and plain linear attention on every high-$V$ benchmark. It must close at least half the gap to softmax iTransformer at linear cost. And its error must degrade measurably more slowly than every baseline as distractors are added.

Why not just…?

Why not just drop cross-series attention and forecast each channel independently? This is a real threat on benchmarks where cross-series information is weak. The synthetic study settles it in principle, because the planted partners carry information the channel-independent model cannot use. On real data, iTransformer's reported average 38.9% improvement over a vanilla Transformer iTransformer indicates that modeling between variates helps. If the channel-independent model ties our layer on PEMS and Traffic, we will report that cross-series mixing at these lookback lengths is mostly spurious. That would be a useful negative finding.

Why not just use softmax attention, which already suppresses background noise? It costs quadratic memory in $V$. It also carries its own failure, the alignment trap proven in High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification, and CGTFra's authors observe it putting weight on coincidental noise when $V$ is large. The distractor test checks this directly. Softmax should draw most of its spurious mass from a few distractors that happen to align, and that mass should grow with $\log(V+M)/L$. If softmax instead stays clean as $M$ grows, the motivation for a linear method narrows to cost alone.

Why not just center the keys, or screen variables with a sparse regression before attention? Centering is the closest cheap competitor and is included as an ablation. It removes the global averaging offset but not chance alignment between two series that merely share a daily cycle, which is the bias the surrogate targets. Results on Weather, where few series share structure, and on PEMS, where nearly all do, will show whether the distinction matters. If centering matches the surrogate everywhere, the method should be simplified to centering and the bias-versus-variance analysis kept as the contribution. Static screening with sparse regression picks one fixed partner set for all windows. It also inherits the false positives that A Selective Overview of Variable Selection in High Dimensional Feature Space attributes to low penalties, and it cannot follow relationships that change from window to window. We include a screened softmax variant on ECL to check this claim.

Decisive Checks

The proposal stands or falls on three outcomes. In the synthetic phase diagram, linear attention's attention mass on non-partners must be driven mainly by shared spectral structure and by $V$, and the surrogate correction must remove most of it while centering does not. In the distractor test on ECL, Traffic and PEMS07, the corrected layer's forecast error must grow more slowly in $M$ than plain linear attention, ELinFormer and softmax iTransformer, at equal training budget. On the unmodified benchmarks, gains must grow with the ratio $V/L$: they should be largest on PEMS07 and Traffic, near zero on low-dimensional Weather and ETT, and should shrink when the lookback is lengthened to 720. A uniform gain unrelated to $V/L$ would mean the layer helps for some reason other than removing chance correlation, and we would report it as such.