Shrinkage Whitening as a Drop-in Normalizer for Multivariate Forecasting

Most deep forecasters for multivariate time series use a small trick to cope with data whose statistics change over time. Before the model sees each input window, it subtracts that window's mean and divides by its standard deviation, one variable at a time. After forecasting, it undoes both steps. This is reversible instance normalization. It works because the model then only has to learn shapes, not levels. It has a blind spot, though: it treats every variable as if it drifted on its own. Real systems rarely behave that way. When a heat wave arrives, temperature and electricity load become more tightly coupled. When traffic starts to jam, neighboring sensors move together. What drifts is not only each variable's scale but also the correlation structure among variables, and per-variable normalization leaves that part of the shift for the network to absorb.

The natural fix is to decorrelate as well as rescale: whiten each window with its own covariance matrix, forecast in the whitened space, and then undo the whitening. Done naively this fails. A lookback window of 96 steps over 321 or 862 variables yields a sample covariance that is singular, and even when it is not singular its small eigenvalues are too small, so inverting it amplifies noise. This note proposes to fix that failure with closed-form covariance shrinkage. For each window, the method blends the sample covariance with the diagonal of per-variable variances. The blending weight is computed analytically for that window, with a correction for the fact that consecutive time steps are not independent. At full shrinkage the layer reduces exactly to per-variable normalization. With no shrinkage it is full whitening. Each window's own statistics decide where it falls between the two, and nothing extra is learned.

Core Claim

A reversible normalization layer that whitens each lookback window with a shrinkage covariance should reduce forecasting error relative to per-variable normalization on datasets where cross-variable correlation varies across windows but persists from lookback to horizon. It should not hurt elsewhere, because the shrinkage weight moves the layer back toward ordinary per-variable normalization when correlations are too noisy to estimate. The layer also scales to hundreds of variables at cost linear in the number of variables, because the shrunk matrix is a diagonal plus a matrix of rank at most the window length.

For a lookback window of length $L$ over $K$ variables, let $S$ be the sample covariance and $D=\mathrm{diag}(S)$. The normalizer uses

$$\hat\Sigma_\alpha=(1-\alpha)\,S+\alpha\,D,\qquad \tilde x_t=\hat\Sigma_\alpha^{-1/2}(x_t-\mu),$$

where $\mu$ is the window mean, $\tilde x_t$ is the whitened input at step $t$, and $\alpha\in[0,1]$ is the shrinkage weight. The weight is the Ledoit–Wolf-type ratio of the estimated variance of the off-diagonal sample-covariance entries to their squared magnitude, which means it is large when those entries are mostly sampling noise. The estimated variance uses the lag-corrected estimator of Covariance shrinkage for autocorrelated data. With $\alpha=1$, $\tilde x$ is exactly the per-variable standardization of RevIN, with its learnable affine parameters kept on top. The forecast is mapped back with $\hat\Sigma_\alpha^{1/2}$ and $\mu$.

Two design details matter. First, the whitening uses the symmetric inverse square root rather than a Cholesky or PCA factor. The symmetric root is the whitening transform that keeps each whitened channel as close as possible to its original variable, so channel-independent backbones such as PatchTST still see channel $k$ as roughly "variable $k$." Second, writing $\hat\Sigma_\alpha=D^{1/2}\big((1-\alpha)R+\alpha I\big)D^{1/2}$, with $R$ the sample correlation matrix of rank at most $L$, lets the inverse square root be computed from an eigendecomposition of an $L\times L$ Gram matrix. That costs $O(KL^2)$ instead of $O(K^3)$, and it is differentiable end to end.

Where This Sits

The motivating evidence comes from the stationarization literature. RevIN showed that per-variable reversible normalization can cut error dramatically. Non-stationary Transformers showed that normalizing too aggressively ("over-stationarization") erases useful context, which it restores by feeding raw statistics back into attention. Dish-TS separated drift inside the lookback window from drift between lookback and horizon, and predicted horizon scale coefficients. IN-Flow replaced the affine normalization with a learned invertible flow. All four normalize each variable's marginal distribution. None addresses shifts in the cross-variable covariance, which is the gap this proposal targets.

The nesting at $\alpha=1$ is also a direct response to the over-stationarization warning. The layer removes correlation structure only as far as the window's data can support, rather than always stripping it.

The closest existing work is Conditionally Whitened Generative Models (CW-Gen). It whitens forecast targets with a full $K\times K$ covariance predicted by a neural head. That head is kept stable by an empirical penalty and pays cubic cost in $K$. It reports winning 80–90% of comparisons on probabilistic metrics over ETTh1, ETTh2, ILI, Weather and Solar. This proposal differs in three ways:

  • It whitens the inputs to deterministic backbones rather than the targets of a diffusion model.
  • Its covariance is a closed-form estimator with no parameters, so there is no head to overfit and no stability penalty to tune.
  • It runs in linear time in $K$.

CW-Gen is therefore a comparison point, and one experiment swaps our estimator into its pipeline.

The method itself comes from the shrinkage literature. The diagonal-target convex combination follows the standard bias–variance argument summarized by Peter Bell et al.. The autocorrelation correction comes from Sutherland et al., who showed that ordinary Ledoit–Wolf sets the shrinkage weight too low on serially correlated data. Their estimator assumes stationarity, and we only apply it within a single short window. The claim is that stationarity inside one window is a much weaker assumption than stationarity across the whole series, and the synthetic experiments test that claim directly. Chen, Wiesel and Hero supply a robust (Tyler-type) shrinkage alternative for heavy-tailed windows. Their fixed-point iteration costs $O(K^3)$ per step, so we treat it as an ablation on small-$K$ datasets rather than as the main method.

KTVGL tracks time-varying sparse precision matrices in tensor series. It aims at recovering graph edges, not at forecasting, and it is not used here.

Research Program

The work has three parts.

Diagnostic. Before any modeling, we measure on each benchmark how much the correlation matrix varies across lookback windows and how much it changes from each lookback to its horizon. The first quantity is the Frobenius spread of shrunk window correlations around the global correlation. The second is the Frobenius distance between lookback and horizon correlations. This produces a falsifiable prediction: gains should appear where the first quantity is large and the second is small. If correlation drift between lookback and horizon dominates, lookback-based de-whitening fails for the same reason Dish-TS identified for scalar statistics. The follow-up in that case is a small head that predicts a horizon-specific $\alpha$, a single scalar per window, rather than a full covariance.

Layer. We implement the layer with the low-rank computation described above and verify that its gradients are numerically stable as $\alpha\to 0$ when $L<K$. We also study, on synthetic data, whether the analytic $\alpha$ tracks the oracle $\alpha$ that minimizes forecast error.

Evaluation. We run the layer across backbones and datasets under matched budgets, as described below.

Baselines

The naive reference point is each backbone trained with no normalization. It measures how much any normalization helps and anchors the scale of the effects.

The primary baseline is per-variable reversible normalization, RevIN, on the same backbone. It is the exact special case $\alpha=1$ of our layer and is the default in current forecasters, so beating it isolates the value of decorrelation. The strongest number the file reports for it is ETTh2 MSE of 0.471 with an N-BEATS backbone at horizon 960, against 6.408 without it. We will reproduce that setting as a sanity check.

The strongest existing normalizers are Dish-TS and IN-Flow. Dish-TS reports over 10% average improvement over RevIN. IN-Flow reports 28.3% and 21.3% average improvements on PatchTST and iTransformer backbones. The file gives only relative figures for both, so we rerun them rather than quote them.

The strongest cross-variable backbone is iTransformer, which applies attention across variables. The file reports a 38.9% average improvement over a vanilla Transformer and a "massive lead" over PatchTST on PEMS. There is no reported result on Lacuna for iTransformer's absolute MSE on ECL or Traffic, so we will measure it ourselves.

The probabilistic comparison is CW-Gen with its learned JMCE covariance. We compare it against the same pipeline with JMCE's covariance replaced by our shrinkage estimate computed on the lookback window. CW-Gen's reported figure is an 80–90% win rate over its base generators. No per-dataset CRPS values are reported on Lacuna.

Three ablations isolate the key ingredient:

  • $\alpha=0$ with a small ridge, which is essentially unregularized whitening.
  • Standard Ledoit–Wolf $\alpha$ without the lag correction.
  • A single global $\alpha$ tuned on validation, which tests whether per-window adaptivity matters.

A Tyler-shrinkage variant runs on ETT, Weather and Exchange only.

Fairness protocol. Every method uses identical data splits, lookback lengths of 96 and 336, and horizons of 96 to 720. Every method gets the same backbone hyperparameters and the same number of tuning trials, 20 per method per dataset. Results are averaged over three seeds. We will count as a win a lower MSE than the best of RevIN, Dish-TS and IN-Flow on the same backbone in a majority of dataset–horizon pairs, with no pair more than 2% worse than RevIN. On CW-Gen, a win means matching CRPS and QICE while improving ProbCorr at lower cost.

Why not just…?

Why not just use a channel-mixing backbone like iTransformer with ordinary RevIN? This is the most serious threat. Attention across variables might learn window-specific correlations on its own. There are reasons to doubt it fully substitutes. Attention sees tokens that have already been standardized per variable, so correlation drift reaches it only indirectly. And iTransformer handles PEMS-scale $K$ by training on random subsets of variables. Still, this is an empirical question. The experiment that settles it is our layer on iTransformer versus RevIN on iTransformer. If the gain there is negligible while it is large on PatchTST, the honest conclusion is that shrinkage whitening mainly helps channel-independent models and cheap backbones. That would still be useful, but it is a narrower claim, and we will report it as such.

Why not just whiten with the sample covariance, or with off-the-shelf Ledoit–Wolf? When $L<K$, as on ECL and Traffic, the sample covariance cannot be inverted. Even when $L>K$, its smallest eigenvalues are biased toward zero, so its inverse inflates noise. Chen, Wiesel and Hero observed this breakdown in anomaly detection. For plain Ledoit–Wolf, Sutherland et al. showed that autocorrelation drives its shrinkage weight too low. Both are ablations here. If uncorrected Ledoit–Wolf matches the lag-corrected version, the autocorrelation argument is unneeded in practice, and we will say so.

Why not just predict the covariance with a network, as CW-Gen does? A learned head can in principle anticipate horizon correlations, which lookback statistics cannot. The costs are cubic scaling and no protection against ill-conditioning when $K$ exceeds the window length. The swap experiment inside CW-Gen tests whether a parameter-free estimator loses anything. If the learned head wins clearly on ILI or Solar, that points to strong lookback-to-horizon correlation drift, and the diagnostic should predict it.

Why not just project onto a few principal components per window? Truncation discards the discarded directions outright rather than damping them. It also changes the channel basis from window to window, which breaks the channel identity that channel-independent backbones depend on. We include a top-$r$ PCA normalizer with $r$ tuned on validation as a secondary baseline.

Proposed Experiments

Synthetic. The first test uses vector autoregressive series with $K$ from 7 to 500. Their correlation matrix rotates either abruptly (regime switches) or slowly, with a controlled lookback-to-horizon mismatch. Here we know the oracle whitening, so we can check two things: that the analytic $\alpha$ tracks the error-minimizing $\alpha$, and that the gain over RevIN grows with rotation strength and shrinks with mismatch. We also test within-window stationarity directly, by inserting change points inside windows and seeing whether the lag-corrected $\alpha$ degrades more gracefully than uncorrected Ledoit–Wolf.

Point forecasting. The real-data evaluation covers ETTh1/h2/m1/m2, Weather, Exchange, ILI, ECL, Solar and Traffic. The backbones are PatchTST, iTransformer and the Non-stationary Transformer. Metrics are MSE and MAE, plus wall-clock time and memory against $K$.

Probabilistic forecasting. This uses CW-Gen's datasets (ETTh1, ETTh2, ILI, Weather, Solar) with CRPS, QICE and ProbCorr.

Decisive checks. Three results would settle the claim. First, across datasets, the improvement over RevIN should correlate positively with the diagnostic's cross-window correlation spread and negatively with its lookback-to-horizon drift. If there is no such relationship, the mechanism story is wrong even if average numbers improve. Second, the per-window adaptive $\alpha$ must beat the best single global $\alpha$; otherwise the shrinkage estimator is doing no work beyond a tuned constant. Third, on Traffic ($K=862$), the layer must run in time comparable to RevIN while unregularized whitening fails outright.