One Regularization Path per Lead Time: Horizon-Stratified Spectral Filtering for Kernel Forecasters

Forecasting models get worse the further ahead they predict. This is called prediction horizon decay. On a global weather benchmark, the best purely data-driven model reaches a 7-day temperature error of 3.09 °C mean absolute error (MAE), against 2.15 for the physics-based numerical forecast (WEATHER-5K). Part of this decay is irreducible, because chaotic weather forgets its initial state. We argue that part of it comes from how models are fitted. A direct multi-horizon forecaster predicts the whole future trajectory in one shot and applies one level and one kind of regularization to every lead time. Near lead times behave like a smooth, highly predictable regression problem. Far lead times behave like a rough, noisy one. Those two problems call for different regularizers.

Operator-valued kernel theory makes this argument precise. An operator-valued kernel is a similarity function whose output is a matrix rather than a number, so a single kernel model can predict a whole vector, here the full future trajectory, while sharing information across its coordinates. Meunier et al. proved that vector-valued kernel ridge regression saturates. Once the target is smooth enough, ridge regression stops improving at the optimal rate. Gradient descent with early stopping does not saturate, and neither does principal component regression. Our proposal turns this into a forecasting method and a test. We train one gradient-descent path of a random-feature operator-valued kernel model and give each lead time its own stopping step. We then check whether horizon decay curves change in the ways the theory predicts.

Core Claim

We make three predictions about direct kernel forecasters on real weather and energy series.

First, the right amount of regularization changes systematically with lead time. Let $\beta(h)$ be the effective smoothness of the target at lead $h$, meaning how well the lead-$h$ value is explained by the dominant patterns in the input, and let $s$ be the eigenvalue-decay exponent of the input kernel. Standard spectral-regularization rates put the optimal regularization at $\lambda \approx n^{-1/(\beta+s)}$, where $n$ is the number of training samples (Meunier et al.). Gradient descent's stopping step plays the role of $1/\lambda$, so the optimal stopping step for lead $h$ should scale as

$$t_h^\star \;\propto\; n^{1/(\beta(h)+s)} .$$

Because predictability falls with lead time, $\beta(h)$ should fall too. The theory therefore says near leads should train longer and far leads should stop earlier.

Second, the kind of regularizer matters, not only its strength. At short leads, where $\beta(h) \ge 2$, ridge regression should lose to gradient descent even when ridge's penalty is tuned separately for each lead. That loss is the saturation effect. At long leads the gap should vanish, because the target is rough and saturation never applies.

Third, coupling lead times through a learned output covariance gives the noisy far leads some of the statistical strength of the well-predicted near leads. This should flatten the far end of the decay curve beyond what any per-lead tuning achieves. In kernel terms, the coupling is a separable kernel $k(x,x')\mathbf{B}$: a scalar similarity $k$ between input windows, multiplied by a positive semidefinite matrix $\mathbf{B}$ over (lead time × variable).

What is new is treating the forecast horizon as an ordered axis along which the regularization itself must vary. We use the shape of the decay curve both as the object we try to improve and as a measurement of how smooth the target is at each lead.

Relation to Prior Work

Several papers supply the method. Spectral-regularization theory supplies the filters and the saturation result we test (Meunier et al.). That work assumes independent samples and one smoothness level shared by all output coordinates. We drop the shared-smoothness assumption, and we handle temporal dependence empirically through blocked data splits. Random features for operator-valued kernels make the model tractable at scale, with roughly $\sqrt{n}\log n$ features sufficing in the well-specified case (Nguyen & Mücke). That work is purely theoretical and covers a single input-to-output map. We use its construction and add a time axis. The vector-valued representer theorem and graph-based output operators (Micchelli & Pontil, Kernels for Multi-task Learning) justify parameterizing $\mathbf{B}$ as a lead-time covariance.

Other papers sit close to the idea and constrain it. Entangled kernels were applied to a weather dataset with 365 daily outputs. They treated those outputs as an unordered multi-task vector and never reported error by lead. We reuse that dataset but stratify every result by lead. Dynamic mode decomposition in vector-valued RKHS extracts slow and fast dynamical modes but never analyzes rollout error. Its intuition that fast modes decorrelate quickly is exactly why far leads should rely only on the top spectral components. Holzleitner et al. aggregate estimators across regularization levels without target labels, but only on static images. We tune by blocked validation instead, and treat their aggregation scheme as a possible substitute when validation labels at long leads are scarce.

Finally, some papers motivate the problem. Dish-TS attributes much of long-horizon decay to the future window's statistics drifting away from the input window's. That mechanism is orthogonal to ours, and we control for it rather than compete with it. Informer and POTR show that predicting the whole horizon at once avoids compounding autoregressive error. Our model is direct in the same sense, so autoregressive drift is not the confound here.

Research Program

The model works as follows. The input is a lookback window of $L$ hourly observations of the five WEATHER-5K variables. The output is the next $H$ hours of the same variables, with $H$ up to 168. The input window is mapped through $D$ random Fourier features of a Gaussian kernel (Random Fourier Features for Operator-Valued Kernels). The output operator $\mathbf{B}$ is either the identity, which treats lead times independently, or a regularized empirical covariance of training targets across lead and variable. We run plain gradient descent on the squared loss. The descent is preconditioned by $\mathbf{B}$, so the output coupling enters the model. Iterates are saved on a log-spaced grid of steps. Lead $h$ reads its prediction from the iterate at step $t_h$, and $t_h$ is chosen on a time-blocked validation fold. All lead times thus share one training run, and the per-lead choice costs only bookkeeping.

The analysis has two parts. For each lead $h$, we fit learning curves (excess validation error against $n$, over five data fractions). From the slope we estimate $\beta(h)$, using the theory's rate exponent $\beta/(\beta+s)$, with $s$ estimated from the empirical decay of the kernel's eigenvalues. We then test whether the selected $t_h$ follows the predicted scaling with $n$ and $\beta(h)$, and whether the ridge-versus-gradient-descent gap opens exactly where $\beta(h)$ exceeds 2.

Baselines

The naive reference point is persistence plus climatology: repeat the last observation, blended toward the station's seasonal mean at long leads. Any method that fails to beat it at a given lead has no skill there.

The within-family control is the same random-feature model trained with one regularizer for all leads. We run it once as ridge regression with a single penalty and once as gradient descent with a single stopping step.

The strongest simple competitor is direct ridge regression with an independently tuned penalty for each lead. This is the classical "direct strategy." It receives the same features, the same data, and a grid as fine as our stopping grid. Our method must beat it at short leads, where saturation predicts a gain. If it does not, the claim that the type of filter matters is false. Principal component regression with a per-lead rank serves as a second filter with no saturation, to check that any gain is not specific to gradient descent.

Two ablations isolate the ingredients. Per-lead stopping with $\mathbf{B}=\mathbf{I}$ isolates the stratification. Learned $\mathbf{B}$ with a single shared stopping step isolates the coupling.

The external references are deep forecasters. PhysicsFormer reports an overall WEATHER-5K temperature MAE of 2.34, with Pyraformer at 2.49. WEATHER-5K reports a best data-driven 7-day temperature MAE of 3.09 and 2.15 for ECMWF-HRES. Since kernel methods are unlikely to beat PhysicsFormer outright, we will report the gap honestly and focus on lead-wise comparisons. The deep models we retrain ourselves (PatchTST, iTransformer, Autoformer) get the same lookback window, the same normalization, and a matched tuning budget measured in GPU-hours. For Dish-TS, on ETT, ECL and Weather, we have no per-dataset MSE values from Lacuna, only its reported average gains of 28.6% (univariate) and 21.9% (multivariate). So we will rerun it rather than quote it.

Why not just…?

Why not just tune a separate ridge penalty for each lead time? This is the real threat, so it is the primary baseline. If saturation is irrelevant on these data, per-lead ridge will match us. That would be a clean negative result about whether Meunier et al.'s theory matters in practice for forecasting. The discriminating evidence is a lead-dependent gap: gradient descent should win where the estimated $\beta(h)$ is at least 2 and tie elsewhere. If instead the gap is uniform across leads or absent, the stratification story fails. The coupling ablation then tells us whether $\mathbf{B}$ alone carries any gain.

Why not just use a strong deep forecaster such as PatchTST or PhysicsFormer? On absolute MAE they may well win. Our claim is mechanistic, though, and it can transfer. Deep forecasters also apply one early-stopping point to all output heads. We will therefore test per-lead checkpoint selection on PatchTST: keep the best validation checkpoint separately for each lead. If this improves PatchTST the way it improves the kernel model, the idea generalizes. If it does nothing, the effect may depend on having a tractable spectrum.

Why not just fix the decay with RevIN or Dish-TS normalization? These normalizations correct the gap between the input window's statistics and the future window's. They do not change how much the model should trust weak input patterns at a far lead. We apply RevIN, and separately Dish-TS, to every method. The proposal succeeds only if the lead-wise gains survive normalization. If normalization removes them, our gains were really a correction of distribution shift.

Proposed Experiments

The main study uses WEATHER-5K, with hourly data from 5,672 stations and splits by year: train on 2014–2021, validate on 2022, test on 2023. We report MAE and RMSE separately at each lead from 1 to 168 hours, and the 99.5th-percentile extreme-event score (SEDI) at 24, 72 and 168 hours. For SEDI, the best data-driven model reaches 14.1% and ECMWF-HRES 37.4% (WEATHER-5K). The 365-output weather dataset from Entangled Kernels provides a small-sample check: few samples, many outputs. There we additionally compare against entangled kernel learning, which models input-dependent output coupling, against our fixed $\mathbf{B}$. ETTh1, ECL and Weather at horizons 96 to 720 test whether the effect holds outside meteorology.

Three outcomes would settle the question. The first is that the selected stopping step $t_h$ decreases with lead $h$ and moves with $n$ as the formula predicts, with the learning-curve estimate of $\beta(h)$ explaining most of the variation. The second is a ridge-versus-gradient-descent gap, at matched per-lead tuning, that is positive and statistically significant at short leads and indistinguishable from zero beyond about 72 hours. The third is that, against single-regularizer training, per-lead stopping combined with coupling lowers the MAE averaged over leads 72 to 168 by more than the variation across random seeds.

If the first two outcomes hold and the third fails, the result is a diagnostic rather than a remedy. If the second fails, saturation is not a meaningful contributor to horizon decay on real forecasting data. Either outcome narrows what kernel theory can say about forecasting.