When Does Data Diversity Pay for Model Complexity in Forecasting?

In long-horizon time series forecasting, a single linear map from past values to future values, which can be solved exactly with least squares, routinely matches or beats Transformers with tens of millions of parameters. In a separate line of work, large forecasting models only show clean scaling behavior when their training data mixes many different sources. This proposal connects the two observations with one testable claim. Most of what a diverse training pool gives a large model is the ability to pick, from context, which of several linear dynamics is in play. If that is true, a cheap closed-form mixture of linear maps should recover most of the large model's advantage over a single linear map. Only a small, measurable remainder should be truly nonlinear.

The practical result is a diagnostic and a baseline. Before training anything deep, one fits linear least-squares models per source and pooled across sources. From these, one can estimate how much a complex model could possibly gain on a given data pool, and whether that gain could be captured without deep learning at all. This addresses an open question in the literature: how "diversity" should be measured beyond sample count, and when it moves the point at which complex models overtake linear ones.

Core Claim

We split the gap between a pooled linear forecaster and a deep model into two parts, each measured on held-out data:

$$ L_{\text{pool}} - L_{\text{deep}} = \underbrace{(L_{\text{pool}} - L_{\text{mix}})}_{\text{linear heterogeneity}} + \underbrace{(L_{\text{mix}} - L_{\text{deep}})}_{\text{nonlinear headroom}} $$

The terms are defined as follows. $L_{\text{pool}}$ is the test mean squared error (MSE) of one instance-normalized ridge regression (least squares with an L2 penalty, solved in closed form) fit to the whole training pool. $L_{\text{mix}}$ is the test MSE of a closed-form mixture of ridge regressions, where a cheap classifier on the look-back window chooses which linear map to apply. $L_{\text{deep}}$ is the test MSE of the best tuned deep model.

The hypothesis is that, as the pool becomes more diverse, the first term grows and the second stays small. Diversity would then be operationally defined as the spread of optimal linear operators across the pool, not as the number of points. The claim fails if, on diverse real pools, deep models keep a large margin over the mixture. That outcome is itself informative, because it would locate the real nonlinear signal.

Grounding in Prior Work

Several papers motivate the claim by showing that linear structure explains most forecasting skill. Are Transformers Effective for Time Series Forecasting? showed that single-layer linear models beat Informer, Autoformer and FEDformer by 20–50%. Revisiting Long-term Time Series Forecasting: An Investigation on Affine Mapping showed that an affine map does the work and that random, untrained feature extractors suffice when combined with reversible instance normalization. Time Series Forecasting Through the Lens of Dynamics showed that good forecasters end in a linear past-to-future layer. Its authors call this layout PRE-DYN: processing blocks first, a terminal linear "dynamics" layer last.

We supply the method from An Analysis of Linear Time Series Forecasting Models. That paper proved DLinear and FITS are the same model class as unconstrained linear regression, and it found closed-form least squares beats SGD-trained linear models. We reuse its closed-form solver as the building block of $L_{\text{pool}}$ and $L_{\text{mix}}$.

The data-diversity side motivates the regime we study. Scaling-laws for Large Time-series Models found that power-law scaling across 38 sources breaks when the source mixture is not preserved. However, it never compared against linear baselines. Chronos: Learning the Language of Time Series found that synthetic pattern variety from random Gaussian-process kernels can beat real target-domain data. Both suggest that variety across series is what large models exploit. Neither asks whether that variety is linearly separable.

Three works constrain the design and are the closest to ours.

Scaling Law for Time Series Forecasting varies data volume by subsampling single datasets, from 10% to 100%. It shows that the optimal look-back window grows with data and shrinks with model size. We therefore tune look-back separately for every model, and we vary diversity at fixed volume, which it does not do.

When Bigger isn't Better relates model tiers to the complexity of individual series. It uses Chebyshev polynomials, electricity, and the S&P 500, and it omits linear neural baselines. Our variable differs: we vary heterogeneity across series in a pool, which a series-level complexity score cannot capture.

TFB supplies the 10-domain benchmark and a fair evaluation protocol that forbids the "drop last" trick. It evaluates fixed models on fixed splits only, with no varying of the training pool.

Research Program

The work has three parts.

A controlled synthetic testbed. Series are generated from random autoregressive linear operators. Two knobs are independent: the dispersion of the operators across series, and an added nonlinearity, such as regime switching driven by the series' own state or a saturating feedback. The ground-truth decomposition is known here, so we can check that $L_{\text{pool}}-L_{\text{mix}}$ tracks operator dispersion and that $L_{\text{mix}}-L_{\text{deep}}$ tracks the nonlinearity knob.

Real pools of controlled diversity. We build pools from the TFB domains and from the sources described in Scaling-laws for Large Time-series Models. Diversity is varied by the number of sources and domains included, with total training points held fixed. We also cover the reverse case: many points from a single source.

The mixture-of-linear estimator. Series windows are clustered on cheap context features, such as normalized spectrum and TFB's trend, seasonality, shifting and transition statistics. One ridge map is solved per cluster. A small classifier routes test windows to a cluster. The number of clusters is chosen by validation. Everything is closed form apart from the k-means step and the routing classifier. As a check, we test whether a pre-training heterogeneity estimate predicts the measured deep-over-pooled gain across pools. That estimate is per-source ridge loss versus pooled ridge loss on a small held-out slice.

Baselines

The naive reference point is persistence, which repeats the last value. Are Transformers Effective for Time Series Forecasting? reports that it beat Transformers on Exchange-Rate. It is run on identical test windows. It mainly guards against pools where nothing is learnable.

The pooled closed-form ridge with instance normalization follows An Analysis of Linear Time Series Forecasting Models. That paper reports it beats SGD-trained linear models in 23 of 32 settings (72%). No per-dataset MSE values for it appear in the file, so we re-run it ourselves. It gets the same look-back grid and a penalty chosen on the same validation split as all other models. It defines $L_{\text{pool}}$.

Per-source ridge, one closed-form map per source, is the oracle-routing version of the mixture. It bounds what linear heterogeneity alone can deliver when source labels are known.

The strongest deep competitors are SAMformer, iTransformer and a PRE-DYN Transformer. SAMformer is a one-layer channel-attention Transformer trained with sharpness-aware minimization. It reports a 14.33% MSE improvement over TSMixer and parity with or up to 7.6% gains over the 314M-parameter MOIRAI. iTransformer reports a 38.9% average gain over the vanilla Transformer. For the PRE-DYN Transformer, adding a linear dynamics layer improved Informer's average normalized TFB score from −0.333 to −0.228 in Time Series Forecasting Through the Lens of Dynamics. We also include a decoder-only model trained on the pool in the style of Scaling-laws for Large Time-series Models.

All deep models get the same pool, the same look-back grid, and an equal hyperparameter-search budget measured in trials. Evaluation follows the TFB protocol. The best of these defines $L_{\text{deep}}$.

The key ablation separates the role of routing from the role of clustering. It runs the mixture with random routing and with cluster counts from 1 to the per-source count.

The proposal "wins" under two conditions. First, the mixture must close at least most of the gap between pooled ridge and the best deep model on high-diversity pools, at a cost of CPU minutes. Second, the heterogeneity estimate must rank pools by deep-model gain with high rank correlation. Where the deep model's margin over the mixture is larger than seed-to-seed variation, that gain is counted as genuine nonlinear headroom and reported as such.

Why not just…?

Why not just use per-dataset ridge regression and stop? Per-dataset ridge is the right answer for single-source benchmarks. It is probably why linear models look so strong on ETT and Weather. It is not available in two cases: zero-shot forecasting on a new series, and pools where many sources each have little data. A pooled model must handle those cases. The mixture is exactly the version of ridge regression that works in those cases. The per-source oracle baseline measures how much is lost by not knowing the source label.

Why not just add a terminal linear layer or sharpness-aware training and declare the gap closed? Those fixes, from Time Series Forecasting Through the Lens of Dynamics and SAMformer, improve deep models on fixed benchmarks. They do not say whether the improved model is doing anything a linear mixture cannot. We include both as the deep competitors. If they beat the mixture only on pools with high measured nonlinearity, our decomposition explains their gains rather than competing with them.

Why not just pretrain on more synthetic variety, as Chronos does? This is a real alternative. Gaussian-process kernels such as linear, periodic and RBF largely generate linear-Gaussian dynamics. Chronos's synthetic success would then be a case of our claim, not a counterexample. We test this in two steps. We first measure the decomposition on KernelSynth-style pools. We then compare zero-shot Chronos against the mixture and pooled ridge fit on each TFB target's training split. No reported result was found on Lacuna for Chronos versus closed-form least squares on TFB. If Chronos wins clearly on high-transition domains, that marks where synthetic variety gives real nonlinear value.

Proposed Experiments

The first decisive check is synthetic. With the nonlinearity knob at zero, $L_{\text{mix}}-L_{\text{deep}}$ should stay near zero for every level of operator dispersion. It should rise with the knob. If deep models beat the mixture even on purely linear heterogeneous data, routing is the bottleneck and the estimator must be redesigned.

The second check is the real-data diversity sweep on TFB and the 38-source pools at fixed volume. We report all three losses, MSE and mean absolute error, by horizon, with look-back tuned per model. This tests whether $L_{\text{pool}}-L_{\text{mix}}$ grows with the number of sources while $L_{\text{mix}}-L_{\text{deep}}$ does not.

The third check correlates the residual headroom with TFB's per-series characteristics, such as transition, shifting and correlation. The purpose is to name the data properties that justify model complexity. We expect cross-channel correlation, the setting where Crossformer excelled in TFB, to be one candidate, since independent-channel linear maps cannot represent it.

The fourth check repeats the decomposition under the low-data setting championed by Time-LLM and disputed by Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis. It uses 5% and 10% of training data and asks whether the mixture matches the reprogrammed language model there.