Exogenous-Analog Residual Gaussian Processes as Sidekicks for Neural Forecasters

Neural forecasters for air quality, traffic and energy fail in a predictable way. They do well on ordinary days and badly when outside conditions change: a wind shift, a holiday, a heat spike. Their uncertainty estimates tend to be most confident exactly when they are most wrong. We propose attaching a cheap, local Gaussian Process (GP) to a frozen neural forecaster. A GP is a Bayesian regression model that predicts from similarity to past data and reports how uncertain it is. This GP models only the forecaster's errors. Its crucial feature is that it judges two past moments as "similar" when their external conditions were similar (wind, traffic, calendar, co-pollutants), not merely when they were close in space and time.

The GP plays two roles at once. When the current external situation resembles past situations where the forecaster erred in a consistent direction, the GP corrects the forecast. When the situation has no close analogs, the GP's variance grows on its own, and the combined forecast widens its intervals. No learned gate is needed to decide how much to trust each model. The proposal treats scalable nearest-neighbor GPs as the core mechanism, exogenous variables as the thing that defines the neighbor metric, and heterogeneous ensembling as the lens. It asks whether a structurally different "sidekick" can fix calibration in continuous spatiotemporal regression without the meta-learner that ruins calibration in stacked ensembles.

Core Claim

The residuals of a strong exogenous-aware or foundation forecaster are not noise. They are structured by external context. A nearest-neighbor GP whose conditioning sets are chosen in a learned, low-dimensional exogenous metric captures this structure, while one using Euclidean space-time distance does not. Adding its posterior to the neural forecast yields better probabilistic accuracy and markedly better interval coverage during exogenous shocks. The overhead is a small fraction of the base model's cost.

Concretely, let $f(x)$ be the frozen forecaster's prediction for a target at location $s$, time $t$ and horizon $h$, and let $r = y - f(x)$ be its residual. We place a GP prior on $r$. The predictive distribution conditions on the $k$ nearest training residuals under the following distance:

$$ d(x, x') = \lVert W_{st}(s,t) - W_{st}(s',t') \rVert^2 + \lVert W_z (z - z') \rVert^2 , $$

Here $z$ is the vector of exogenous variables known at forecast time (forecasted weather, calendar, traffic, co-pollutants). $W_{st}$ and $W_z$ are linear maps, with $W_z$ projecting to $q \approx 4$–$8$ dimensions. Both are learned as kernel lengthscale/projection hyperparameters on a small random subset.

The final forecast is Gaussian. Its mean is $f(x) + \mu_r(x)$ and its variance is $\sigma_r^2(x)$, the GP's local posterior variance after calibration. The low-dimensional projection is what keeps neighbor search fast despite many exogenous channels, which is the curse-of-dimensionality concern raised for GPnn.

Relation to Prior Work

Work that supplies the method. GPnn scales GP regression to millions of points by estimating hyperparameters on a subset and predicting each test point from its $m$ nearest neighbors. It reports about 30 seconds of training on $1.6\times10^6$ points on a laptop, and includes a cheap noise-calibration post-process. VNNGP gets similar locality through a sparse precision matrix built from nearest-neighbor conditioning. Both papers use isotropic Euclidean neighbors over raw coordinates, and VNNGP names this as a failure mode for non-stationary data. We reuse GPnn's decoupled estimate-then-predict recipe and change two things: the target becomes a neural model's residual, and the neighbor metric becomes exogenous.

Work that motivates the proposal. Two papers motivate the neural side. ExoST shows that gated selection of exogenous variables plus a balancer between past and future covariate branches improves Mean Relative Error by over 20% on Madrid air quality and traffic. ChronosX shows that adapters injecting covariates into frozen foundation models improve weighted quantile loss and MASE by 22% on its synthetic suite. Neither method gives uncertainty that is grounded in how novel the current exogenous state is. ExoST is deterministic, and ChronosX relies on the base model's quantile head.

Air Quality Arena also motivates the work. It finds that top univariate foundation models struggle on reactive pollutants and in volatile regions, where even supervised models fail to beat Seasonal Naive. Those are exactly the regimes where a local correction with honest variance should help.

Work that constrains the design. Ensembling Tabular Foundation Models constrains the ensembling choice. Similar-prior models produce near-redundant errors (Q-statistic 0.961), and stacking meta-learners degrade calibration. This is why we use an additive Bayesian residual model rather than a learned combiner. Asymmetric Duos shows that a sidekick using about 10% of the base model's FLOPs can match a 5-model deep ensemble's uncertainty quality on image classification, using static temperature weights. It flags input-dependent weighting as open. Our sidekick is input-dependent by construction, because its variance is local, and it operates in continuous regression rather than classification.

Heterogeneous Multi-output GP Prediction uses "heterogeneous" to mean mixed likelihood types. It is not a neural/kernel hybrid and does not use nearest-neighbor scaling.

Research Program

We will build the residual GP on top of three kinds of frozen base forecasters:

  • a spatiotemporal graph network with ExoST (GraphWaveNet backbone) on the Madrid suite;
  • ChronosX-style covariate-adapted foundation models on the M5 and energy collection and on the synthetic covariate suite;
  • the leading univariate foundation models on AQA-Bench, using covariates the dataset itself provides: the other five pollutants at the same station, calendar features, and neighboring stations.

Residuals are computed out-of-fold on the training period, so that the GP never sees in-sample residuals of the model it corrects.

Hyperparameters. The projection $W_z$, the spatiotemporal lengthscales and the noise level are fit by exact GP marginal likelihood on random subsets of a few thousand points, following GPnn. We also try a variant that fits $W_z$ by maximizing held-out predictive log-likelihood of residuals.

Calibration. GPnn establishes only weak calibration, meaning average variance matches average squared error, and environmental shocks are heavy-tailed. We therefore compare the Gaussian posterior against a Student-t predictive with the same mean and scale. This is an explicit study of whether strong, quantile-level calibration holds, not an assumption that it does.

Analysis. We measure how much residual variance is explained by exogenous-metric neighbors compared with Euclidean ones. We relate the GP variance to a direct novelty score, the distance to the $k$-th exogenous neighbor, to test whether variance inflation tracks genuinely unseen conditions.

Baselines

Base forecaster alone. This is the naive reference point: the frozen base forecaster with its own uncertainty. That means ExoST on GraphWaveNet for Madrid, ChronosX-adapted Chronos for M5/energy and synthetic, and the top AQA zero-shot model (VisionTS++, reported rank 1 on MASE across seven countries in Air Quality Arena). Because ExoST is deterministic, we give it a Gaussian predictive with variance equal to its validation mean squared error.

Seasonal Naive is also reported, because Air Quality Arena shows it is competitive in India and Mexico.

Strongest existing exogenous-aware methods. These are ExoST itself, with its reported >20% MRE improvement over target-centric models and roughly 10% gain on GraphWaveNet on Madrid, and ChronosX, with its reported 22% WQL/MASE improvement over zero-shot Chronos on the 32-task synthetic suite. We rerun both on our splits rather than quoting their numbers as targets, giving them the same data, covariates and tuning budget in GPU-hours as our full pipeline. Beating them means lower continuous ranked probability score (CRPS) and weighted quantile loss with no worse MASE. It also means 90% interval coverage within two points of nominal on the shock subsets, which we define below.

Ablation of the key ingredient. This is the identical residual GP with neighbors chosen by Euclidean space-time distance only: vanilla GPnn applied to residuals, which is classical regression kriging. If this matches the exogenous-metric version, the central claim fails.

Static duo. This is an Asymmetric-Duo-style combination: an independently trained GPnn predicting the target directly, mixed with the neural forecast using a single learned weight per dataset.

Stacking. This is a learned stacking meta-model that takes both predictions and exogenous features as inputs. It tests the calibration trap in the regression setting.

Why not just…?

Why not just feed the exogenous variables as extra inputs to a single GP or to the neural model? Putting them in the neural model is exactly what ExoST and ChronosX already do, and they are our base models. The question is whether structured error remains after doing so. If the residual GP gives no gain on top of ExoST or ChronosX, that is a clean negative result, and the experiment is designed to report it.

A single GP on concatenated coordinates and exogenous features, without the neural mean, is included as a baseline. Prior evidence that it will lose is that the ExoST paper finds naive concatenation of uncurated covariates hurts.

Why not just recalibrate the neural forecaster with conformal prediction or a quantile model on residuals, conditioned on exogenous features? This is the real threat. A gradient-boosted quantile model on residuals could capture the same context-dependent error widths. We will run it with the same out-of-fold residuals and features.

Our expected advantage is specific. Such a model extrapolates arbitrarily in unseen exogenous states, while the GP's variance reverts to the prior variance by construction. The decisive comparison is therefore coverage on the held-out shock subsets, not average CRPS. If the quantile model matches the GP there, we would recommend the simpler model.

Why not just ensemble several neural or foundation forecasters? The tabular foundation model audit finds near-redundant errors and only a +0.18% gain at 253× compute for similar-prior ensembles. We still run a five-seed deep ensemble of the base forecaster as a compute-heavy reference, and report CRPS per unit of inference cost.

Proposed Experiments and Decisive Checks

Benchmarks. The benchmarks are the Madrid suite (AQI-19, AQI-22, Speed-19, Intensity-22), the ChronosX synthetic suite, the M5 and energy collection, and AQA-Bench with its standard 168-hour lookback and 24-hour horizon. We found no reported numerical CRPS or coverage results on Lacuna for any of these, so all probabilistic numbers will be produced by our reruns.

Shock subsets. The most informative check is controlled. On the synthetic suite we hold out every instance of one shock family, either spikes or steps, so that its exogenous signature never appears in training residuals. We then measure whether the exogenous-metric GP widens its intervals there while the Euclidean ablation and the quantile recalibrator do not.

For the real data, shock subsets are defined beforehand: holidays and top-5% wind-change hours for Madrid; India (CPCB) and Mexico (SINAICA) stations and reactive pollutants for AQA-Bench.

Metrics. We report CRPS and weighted quantile loss, MASE and MRE, 50% and 90% interval coverage, and probability integral transform histograms. Together these test strong calibration rather than only matching average variance to average error.

Overhead. We measure wall-clock overhead relative to the base model's forward pass, against the roughly 10–20% budget that made Asymmetric Duos practical.

Success criteria. The proposal succeeds if the exogenous-metric residual GP beats the Euclidean ablation on CRPS on the shock subsets of at least three of the four benchmark families. It must also match or beat conformal and quantile recalibration on shock-subset coverage, and stay within the 20% overhead budget.