Residual-Conditioned Exogenous Selection for Time-Series Foundation Models

Pretrained time-series foundation models are large forecasters that predict a series from its own history without any task-specific training. On air-quality forecasting across seven countries, these models already beat custom supervised models that are given many extra input variables. Yet they still fail where outside drivers matter most. We propose to stop asking which external variables predict the target, and to ask instead which external variables explain what the foundation model gets wrong. Our method keeps a frozen univariate foundation model as the forecaster. It then selects a small set of external variables (weather, calendar, neighbouring sensors, co-measured pollutants) and fits a light correction model that predicts the foundation model's error. The variable choice and the correction model's settings are tuned together.

The central bet concerns redundancy. Most candidate external variables are redundant in two ways. They duplicate each other: dew point, humidity and temperature move together, as do PM10 and PM2.5. They also duplicate information the target's own history already carries, such as daily cycles and weekly periodicity. Existing selectors measure only the first kind, and only against the raw target. If we measure relevance against the foundation model's residual, after removing the predictable part of each external series, the candidate pool should shrink from dozens or hundreds of variables to a handful. That makes the joint search over variables and correction-model settings both cheap and less prone to overfitting.

Core Claim

Exogenous inputs, meaning variables external to the target series, help only through their unpredictable part. That part must also be aligned with the unpredictable part of the target. Write $r_t = y_t - \hat y_t$ for the residual of the frozen foundation model's forecast $\hat y_t$ of the target $y_t$. Write $u_{j,t}$ for the innovation of candidate variable $x_j$, meaning $x_j$ minus a cheap forecast of $x_j$ from its own past. Given an already selected set $S$ of variable–lag pairs, we score candidate $j$ at lag $\ell$ by

$$ s(j,\ell \mid S) = \mathrm{HSIC}\big(r_t - g_S(\cdot),\; u_{j,t-\ell}\big), $$

where HSIC (the Hilbert–Schmidt Independence Criterion) is a kernel measure of statistical dependence. Here $g_S$ is the correction model already fitted on $S$. Candidates are added greedily while the score clears a permutation-calibrated threshold. The surviving pool, not the raw feature list, defines the binary part of a joint search over the variable mask and the correction model's hyperparameters.

The claim is falsifiable in three ways. Residual-conditioned screening should shrink the candidate pool much more than marginal redundancy screening does. It should match or beat full-pool joint search at a fraction of the evaluations. And it should turn a foundation model that cannot beat a seasonal-naive forecast in volatile regions into one that clearly does.

Where This Sits

The proposal is motivated by a contradiction in the literature. Exogenous-aware spatio-temporal networks report clear gains over history-only models. ST-ResNet fuses weather and holidays through a dedicated subnetwork and reaches RMSE 16.89 on TaxiBJ against 18.18 for DeepST. UrbanFM embeds external factors at several stages, and Aura routes sensor, metadata and text streams through cross-attention and expert sub-networks. In contrast, Air Quality Arena finds zero-shot univariate models such as VisionTS++ and TiRex beating supervised multivariate models including PatchTST. It also finds every model struggling in India and Mexico, where the foundation models barely beat seasonal naive. Our reading is that the exogenous signal is real but sparse, and drowned in redundant inputs. That reading predicts both findings.

The method for joint selection comes from two lines of work. Ambulance Demand Prediction via CNNs treats exogenous inclusion flags as categorical hyperparameters inside Bayesian optimization, a sample-efficient search that fits a model of the score surface. It uses dimension-dropout and hierarchical variants, and reports a 40.4% parameter reduction with under 0.05% loss. It treats candidates as independent and selects against the raw target. Integrated Evolutionary Learning and CIFOA-SVM supply the argument that feature subsets and model settings are coupled. Both work on static tabular data. We reuse their joint formulation but change what the binary part of the search is chosen against.

The redundancy machinery comes from the screening literature, which also constrains our design. SmRMR turns minimum-redundancy-maximum-relevance into penalized estimation with HSIC and knockoffs. It warns that knockoffs become unstable at small sample sizes, so we use permutation thresholds on time-blocked data rather than knockoffs. RCDFS shows that one early false selection distorts later greedy steps. We address this by re-fitting $g_S$ before each scoring step. Greedy eigencomponent selection shows that variance-based reduction keeps the wrong directions, and so rules out PCA on exogenous inputs as a shortcut. Repulsive selection by determinantal point processes, sampling methods that favour mutually dissimilar subsets (column-subset DPP, DPP priors), handles redundancy among candidates. It is blind to redundancy with the target's own history, which is the kind we expect to dominate.

Research Program

We will build a two-stage system. Stage one runs the frozen foundation model, VisionTS++ or TiRex, over training windows to produce residuals. It fits a cheap per-variable autoregressive forecaster to obtain innovations, and runs residual-conditioned greedy screening over variable–lag pairs. The correction model $g$ is a small gradient-boosted tree model or a two-layer network. It receives the selected innovations plus known-future calendar flags. Exogenous values are only ever taken from the past, or from known-future calendar fields, so there is no leakage.

Stage two runs joint Bayesian optimization, following the hierarchical design of the Ambulance paper. The search covers the mask over the screened pool, the lag set, the correction model's capacity and regularization, and the blend weight between $\hat y_t$ and the correction. The screening threshold is itself a searched hyperparameter, so the joint loop can loosen screening if it was too aggressive.

We will also test the coupling assumption directly, since it underpins all joint-optimization work. We will measure how stable the selected subset is, by Jaccard overlap, across correction-model capacities and across the foundation model's context lengths. If subsets are invariant, sequential selection-then-tuning is sufficient. We would report that as a result rather than hide it.

Baselines

The naive reference points are seasonal naive and the frozen foundation model alone, VisionTS++ and TiRex at the benchmark's 168-hour context. Air Quality Arena reports VisionTS++ with the lowest MASE (mean absolute scaled error) in almost every country. It gives no exact numeric values in the file, so for those tables there is no reported result found on Lacuna. We will re-run both models ourselves on identical splits.

The strongest exogenous-aware supervised competitors are PatchTST, the only supervised model competitive in Air Quality Arena, and on urban flows ST-ResNet. ST-ResNet reports RMSE 16.89 on TaxiBJ and 6.33 on BikeNYC. Beating it on TaxiBJ means a lower RMSE than 16.89 on the same test split, with a seed-variance interval excluding it.

The strongest joint-selection competitor is Ambulance-style hierarchical Bayesian optimization run over the full unscreened candidate pool against the same residual target. It gets the same number of correction-model fits and the same correction-model family as our method. It isolates whether screening adds anything beyond the residual reformulation.

Two ablations isolate the key ingredient. In the first, screening scores against the raw target $y_t$ instead of the residual $r_t$. In the second, screening uses raw $x_j$ instead of the innovation $u_j$. Each also gets the identical evaluation budget.

Beating these means three things together: lower error than the foundation model alone in the volatile regions, equal or lower error than full-pool joint search at no more than a quarter of its evaluations, and a smaller selected set than both ablations.

Why not just…?

Why not just cluster correlated exogenous variables and pick one per cluster, then tune normally? This removes redundancy among candidates cheaply, and it is a real competitor on datasets whose candidates are a few weather fields. It cannot remove redundancy with the target's history, for example a temperature cycle that the foundation model already captures. Its cluster representative is also chosen without regard to the residual. We run correlation clustering followed by Bayesian optimization as a baseline. If it matches our method on AQA, where neighbouring stations and co-pollutants create large collinear pools, we would conclude the residual conditioning is unnecessary.

Why not just feed every exogenous variable into a gated subnetwork and let training sort it out? ST-ResNet and UrbanFM do exactly this and gain over history-only models. The Air Quality Arena results suggest that such supervised multivariate models lose to zero-shot univariate ones once the series are diverse. We include a full-input gated correction model on the same residual target. A clear win for it would mean explicit selection is redundant with learned gating.

Why not just fit a LASSO correction on all exogenous innovations? This is the simplest residual-stacking alternative and the most serious threat. SmRMR reports that L1 penalties fail to separate groups of correlated predictors and select about three times more features than its non-convex variant. That evidence is from static data. We run LASSO and elastic-net corrections with their penalty tuned under the same budget. If they match our accuracy, the contribution narrows to interpretability and subset size, and we will say so.

Why not just use the foundation model alone? In the volatile Indian and Mexican series of Air Quality Arena it barely beats seasonal naive. That is precisely the regime where we expect exogenous innovations to matter, and where gains should concentrate.

Proposed Experiments

The primary benchmark is AQA-Bench, with 168-hour context and a 24-hour horizon. For each PM2.5 or O3 target series, the candidate pool is co-located pollutants plus the twenty nearest stations' series, giving well over a hundred heavily redundant candidates. We report MASE per country, with CPCB and SINAICA as the regions where the claim must hold.

The secondary benchmarks are TaxiBJ and BikeNYC. Here the foundation model forecasts each grid cell's flow, and candidates are the dataset's weather and holiday fields plus neighbouring-cell flows. RMSE is compared to ST-ResNet's reported 16.89 and 6.33. The Seattle ambulance data would be a natural third testbed if accessible. Its reported more-than-9% MSE gain over the operational Medic method gives a reference point, though not a strict baseline.

Three checks decide the proposal. First, screened-pool joint search must reach the error of full-pool joint search with at least four times fewer correction-model fits. If not, screening is just a speedup with an accuracy cost. Second, the residual-conditioned ablation must beat raw-target screening specifically in the volatile regions. If it does not, our redundancy hypothesis for the foundation-model-versus-multivariate contradiction is wrong. Third, the subset-stability analysis tells us whether joint optimization was needed at all, or whether a good screen followed by ordinary tuning suffices.