Telling Surges from Glitches: Robust Low-Rank Decomposition with Corrupted Exogenous Drivers

Sensor networks for traffic, electricity and air quality produce tables of readings: time along one axis, locations along the other. Two communities treat the large deviations in these tables in opposite ways. Forecasting researchers add exogenous variables, meaning outside signals such as weather, holidays or scheduled events, so that a model can predict sudden surges. Robust dimensionality-reduction researchers fit a smooth, low-rank summary of the table and discard any entry that deviates sharply as "sparse corruption." Applied to a city grid, the second approach deletes the heatwave load spike that the first approach is trying to forecast. Neither approach handles the case where the outside signal is itself broken, for example a stuck weather station or a mislogged event.

We propose one decomposition that splits a spatio-temporal table into three parts: routine low-rank structure, surges that the exogenous series explain, and isolated sensor glitches. The exogenous series is cleaned inside the same fit, so a glitch in a weather feed cannot be passed off as an explanation for a glitch in the target. The rule that separates the parts is simple. A deviation counts as a real surge only if a cleaned outside driver, at some plausible time lag, accounts for it coherently across many locations. Otherwise the deviation is treated as a glitch. The cleaned output then feeds any downstream forecaster.

Core Claim

Exogenous context can act as a certificate for the split between "low-rank plus explained" and "sparse outlier." Joint estimation matters in the regime practitioners actually face, where both the target and the covariates carry gross outliers. In that regime the obvious sequential pipelines fail, and they fail in specific ways. Blind robust PCA (principal component analysis that tolerates outliers) removes real surges. Regressing on the covariates first and then running robust PCA passes covariate glitches into the regression. We expect the joint model to keep true surges and reject both kinds of glitch, and we expect that difference to show up as lower forecasting error at peak hours.

Prior Work and Positioning

The robust-subspace side supplies the machinery. Principal Component Pursuit writes a table as low-rank plus sparse, and Stable Principal Component Pursuit shows that recovery stays bounded when dense noise is added. Both treat every large residual as corruption, and this is exactly the behaviour that erases surges. Robust PCA: Optimization of the Robust Reconstruction Error over the Stiefel Manifold trims the largest errors instead and is fast. Trimming has the same blindness to context. Robust Principal Component Analysis on Graphs adds a fixed affinity graph as a smoothness prior. Its context is static and cannot say why a deviation happened.

The closest work is the side-information line. Robust Principal Component Analysis with Side Information (PCPF) forces the low-rank part to be a bilinear map of fixed feature matrices. It proves that this raises the recoverable rank, but it requires the true subspace to lie inside the feature span. Using Side Information to Reliably Learn Low-Rank Matrices from Missing and Corrupted Observations relaxes that requirement by adding a residual low-rank term. It tolerates Gaussian noise in the features but assumes gross outliers live only in the target. It also treats the features as static row and column descriptors, not as time series with lags.

Our proposal changes three things relative to these papers. The side information becomes a lagged exogenous time series. The side information gets its own sparse corruption term. The exogenous-explained part is kept separate from the routine low-rank part so it can be read off as "surge."

The forecasting side supplies motivation and downstream consumers. TimeXer integrates exogenous series through cross-attention and reports that it under-predicts the height of sudden spikes on Traffic. ChronosX adds covariate adapters to frozen pretrained forecasters. Predicting Citywide Crowd Flows Using Deep Spatio-Temporal Residual Networks (ST-ResNet) fuses weather and holiday features into crowd-flow prediction. All three consume covariates as given and never ask whether a covariate reading is itself an outlier. We do not replace these models. We give them a cleaner input and an explicit surge channel.

Model

Let $Y \in \mathbb{R}^{T \times n}$ be the target readings over $T$ time steps and $n$ locations. Let $Z \in \mathbb{R}^{T \times m}$ be the observed exogenous series. Let $D_\ell$ denote the operator that shifts a series down by $\ell$ steps. We model $Z = Z^\circ + S_Z$, where $Z^\circ$ is the clean exogenous signal and $S_Z$ is sparse, and $Y = L + \sum_{\ell=0}^{K} D_\ell Z^\circ B_\ell + S_Y + E$. Here $L$ is routine low-rank structure, $B_\ell \in \mathbb{R}^{m \times n}$ maps lag-$\ell$ drivers to locations, $S_Y$ is sparse target corruption, and $E$ is dense noise. We estimate all of these by solving

$$\min \; \tfrac12\Big\|Y - L - \sum_{\ell} D_\ell Z^\circ B_\ell - S_Y\Big\|_F^2 + \|L\|_* + \alpha \sum_\ell \|B_\ell\|_* + \lambda_Y\|S_Y\|_1 + \lambda_Z\|S_Z\|_1 + \beta\, \mathcal{R}(Z^\circ)$$

subject to $Z = Z^\circ + S_Z$. In this objective, $\|\cdot\|_*$ is the nuclear norm (the sum of singular values, a convex stand-in for rank), and $\|\cdot\|_1$ is the entrywise absolute sum. $\mathcal{R}$ is a prior on the clean covariates: a nuclear norm when many correlated weather stations are available, or temporal smoothness when only a few series exist.

The nuclear penalty on $B_\ell$ matters for identifiability. It forces a single exogenous driver to act through a few shared spatial patterns. A heatwave therefore raises load coherently across a region, whereas a glitch at one meter cannot borrow that explanation. The covariate term $S_Z$ plays the matching role on the other side. A stuck temperature reading contradicts both the other stations and the absence of any target response, so paying $\lambda_Z$ to remove it is cheaper than fitting it.

The problem is bilinear in $Z^\circ$ and $B_\ell$. We solve it by alternating minimization. Each block is either a proximal step (a shrinkage step that sets small entries to zero, used for the sparse terms) or a factored low-rank update, $L = UV^\top$ with fixed rank. The factored update avoids the full singular value decompositions that make convex robust PCA slow. The result can be applied on sliding windows.

Research Program

The work has four parts, in order.

First, we build the solver and a synthetic generator. The generator extends the ChronosX spike, step and bell construction by injecting sparse outliers into the target and into the covariates at controlled rates.

Second, we map phase diagrams over target corruption rate, covariate corruption rate, and surge amplitude, in the spirit of PCPF's "gray region" analysis. The question is where each competitor breaks.

Third, we study identifiability. We ask under what spatial-coherence conditions on $B_\ell$ a surge cannot be absorbed by $S_Y$, analogous to the incoherence conditions in PCPF. We will also report when the method must fail: surges that are spatially isolated but real look like glitches by construction.

Fourth, we test on real data, feeding the cleaned tables and the extracted surge channel to existing forecasters. We will choose the $\lambda$ parameters by predicting held-out masked entries. This is a label-free criterion and responds to the open tuning problem for robust PCA.

Baselines

The naive reference is running the forecaster on raw data with its covariates as given. We will also report the Seasonal Naive forecast, which Air Quality Arena found hard to beat in India's monitoring network.

The first cleaning baseline is blind robust PCA (stable Principal Component Pursuit) and trimmed robust PCA, applied to $Y$ before forecasting. These represent standard outlier removal.

The strongest structural competitor is the noisy-side-information model from Using Side Information to Reliably Learn Low-Rank Matrices from Missing and Corrupted Observations. We will give it lagged exogenous series as side features, which is the most charitable adaptation. PCPF itself runs as a secondary baseline. Its reported numbers are on image data: about 87.8% versus 87.0% SVM accuracy against blind Principal Component Pursuit on MNIST at 0.3 corruption. That establishes that side information helps but says nothing about temporal covariates. Beating it means higher surge retention and outlier-detection F1 at matched runtime.

The pipeline we consider the real threat is robust regression of $Y$ on lagged $Z$ followed by robust PCA on the residuals. It is described below.

On the forecasting side, TimeXer and ChronosX run on the same splits, look-back windows and tuning budgets, with and without our cleaning. We found no reported Lacuna result for TimeXer's per-horizon numbers on its electricity-price (EPF) datasets. For ChronosX the file reports only a 22% improvement in weighted quantile loss and mean absolute scaled error (MASE) over zero-shot Chronos on synthetic data. On TaxiBJ, ST-ResNet reports an RMSE of 16.89, against 18.18 for DeepST and 22.78 for ARIMA. That 16.89 is our reference, and our claim is judged mainly on peak-hour RMSE after contamination.

Ablations isolate each ingredient: removing $S_Z$, removing the nuclear penalty on $B_\ell$, and setting $K=0$ so that no lags are allowed.

Why not just…?

Why not just run robust PCA first and then forecast with TimeXer? Blind robust PCA has no way to tell a heatwave from a glitch, and large deviations are exactly what its sparse term absorbs. We will measure this directly: the fraction of injected, exogenously caused surge energy that survives outside $S_Y$. We expect it to be low for blind methods, and PCPF's gray-region results suggest blind recovery collapses as structure gets richer.

Why not just regress on the covariates first, then run robust PCA on the residuals? This is the real threat. When covariates are clean and effects are roughly linear, it should match our method, and we will say so if it does. Our method differs in one regime. When $Z$ contains glitches, the first stage fits them, even with a robust loss, because the loss protects against target outliers, not against corrupted regressors. The covariate glitches then appear as spurious "explained surges." The phase diagram over covariate corruption rate decides the question. If the sequential pipeline stays within noise of ours at every covariate corruption rate, the joint formulation is not justified, and we will narrow the claim to the lag-coupled, low-rank loadings.

Why not just train TimeXer or ChronosX with a Huber loss on raw data? A robust loss down-weights large errors. For surges this is the wrong direction, and it would likely worsen the peak under-prediction TimeXer already reports. The loss also does nothing about corrupted covariate tokens entering cross-attention. We test this by training both models with a Huber loss on raw contaminated data and comparing peak-hour error against the same models fed our cleaned tables.

Why not just use the noisy-side-information model as is? It assumes gross outliers never occur in the features, and it has no notion of lag. We give it lagged features to remove the second handicap, so any remaining gap is attributable to the covariate corruption term.

Proposed Experiments

Synthetic. We extend the ChronosX construction to 200 to 500 locations sharing regional exogenous drivers. We inject target outliers at 1–20% and covariate outliers at 0–10%. Metrics are surge retention, outlier-detection F1 separately for $S_Y$ and $S_Z$, and recovery error of $L$.

TaxiBJ. We use the weather and holiday covariates from ST-ResNet. We inject synthetic sensor glitches into flows and weather, then evaluate downstream RMSE overall and on the top 1% of hours by flow.

TimeXer suite. On EPF and Electricity Consuming Load (ECL), the exogenous load and generation forecasts are natural drivers. We corrupt them at controlled rates and evaluate MSE and MAE, plus peak MSE on the Traffic dataset where TimeXer reports its spike problem.

Air Quality Arena. We pair a subset of stations with meteorological covariates. Air Quality Arena notes that such covariates are absent from the benchmark, so this pairing is our own data work, and results would be reported as exploratory. Air Quality Arena reports no numerical error tables on Lacuna, so we compare MASE against Seasonal Naive and the published top-tier ranking.

Decisive checks. Three outcomes decide whether the proposal holds up. The first is whether, at nonzero covariate corruption, our method beats robust-regression-then-robust-PCA on surge retention and detection F1. The second is whether removing $S_Z$ erases that gain, which would identify covariate cleaning as the source. The third is whether cleaned inputs lower peak-hour error for TimeXer and ChronosX without raising overall MAE. If the first check fails, the idea reduces to a known pipeline and we will report that.