The Linear Model Paradox in Time Series Forecasting

15 concepts13 papers1 proposal

In deep learning, the dominant paradigm is clear: larger architectures with more parameters and complex attention mechanisms yield superior performance. This holds true across natural language processing and computer vision. However, long-term time series forecasting (LTSF) presents a striking anomaly. In this domain, highly sophisticated architectures frequently underperform relative to "embarrassingly simple" linear models and Multi-Layer Perceptrons (MLPs).

Understanding this complexity-performance paradox requires looking closely at the structural properties of temporal data, the pitfalls of over-parameterization, and the specific inductive biases required to map the past to the future.

The Empirical Discovery of the Complexity Gap

The tension between model complexity and forecasting accuracy came to the forefront when Ailing Zeng et al. published Are Transformers Effective for Time Series Forecasting?. The authors demonstrated that a family of single-layer linear models (LTSF-Linear) consistently outperformed complex, state-of-the-art Transformer architectures on standard benchmarks, often by margins as large as 20% to 50%.

This finding challenged the assumption that self-attention mechanisms designed for natural language are inherently suited for numerical sequences. Rather than extracting deep semantic features, the heavy lifting in successful neural forecasters is often driven by simple affine mappings. As shown by Zhe Li et al. in Revisiting Long-term Time Series Forecasting: An Investigation on Affine Mapping, these linear mappings are exceptionally proficient at capturing periodic patterns. When the weights of complex feature extractors are left unoptimized or over-engineered, their contribution to the actual forecast is minimal, raising fundamental questions about the necessity of intricate designs.

Why Complexity Fails: Information Density and Overfitting

Why do massive architectures struggle where a single linear layer excels? The answer lies primarily in the difference in information density and the risk of overfitting.

Low Information Density

Natural language tokens carry rich, discrete semantic meaning. In contrast, continuous numerical time series data has a much lower information density. When massive models are applied to these sequences, their high parameter count becomes a liability. Liangwei Nathan Zheng et al. explored this in Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis: The Impact of Modality Alignment, demonstrating that 32-layer Large Language Models often fail to outperform simple linear models because their massive capacity leads to overfitting rather than the discovery of true temporal patterns. This over-parameterization trap also impacts generative architectures; Jiaxin Gao et al. observed in Auto-Regressive Moving Diffusion Models for Time Series Forecasting that using heavy Transformer backbones as diffusion operators introduces unnecessary complexity, degrading performance on smaller datasets.

Temporal Distribution Shifts

Complex models are highly adept at memorizing the specific quirks of their training data. In time series, however, environments evolve due to shifting demographics, policy changes, or seasonal variations. Helen Zhou et al. highlighted this vulnerability in Evaluating Model Performance in Medical Datasets Over Time. While high-capacity models (such as gradient-boosted trees) perform exceptionally well on contemporary data, they exhibit lower robustness to temporal distribution shifts. As the gap between training and deployment increases, simpler linear models eventually surpass them because they do not overfit to era-specific features.

Mismatched Inductive Biases

Many time series originate from deterministic, well-understood physical or economic processes. Nga Yu Lo et al. demonstrated in When Bigger isn't Better: The Role of Model-Data Complexity in Time Series Forecasts that while massive foundation models can handle highly intricate, noisy data, they underperform on basic, deterministic time series. They fail to leverage their capacity for simple patterns where classical statistical models with appropriate inductive biases naturally thrive.

The Mechanics of Success: Direct Mapping and Learning Dynamics

The success of linear models is not accidental; it is rooted in how they preserve and map temporal structures.

A primary reason complex sequence models fail is their lack of specific structural priors required to map historical sequences directly to future values. This concept is formalized through the PRO-DYN framework by Alexis-Raja Brachet et al. in Dynamics is what you need for time-series forecasting!. The authors argue that a model's forecasting success depends heavily on having a learnable block at the very end of the architecture that directly maps past observations to future predictions.

Direct Linear Mapping (LTSF-Linear / PRO-DYN):
[Past Horizon: X_1, ..., X_t] ----(Learnable Linear Weights)----> [Future Forecast: Y_1, ..., Y_h]

Indirect Mapping (Standard Transformer):
[Past Horizon] --> [Temporal Embedding] --> [Self-Attention Layers] --> [Projection] --> [Future Forecast]

When architectures lack this direct mapping, they struggle to preserve the underlying data dynamics. In Time Series Forecasting Through the Lens of Dynamics, the authors show that simply appending a learnable linear dynamics layer to the end of underperforming, complex models can significantly boost their accuracy, proving that learning direct temporal dynamics is the missing key.

Furthermore, the relationship between the historical context window and model complexity is tightly constrained. As proven by Jingzhe Shi et al. in Scaling Law for Time Series Forecasting, scaling laws do exist in time series forecasting, but they are governed by an optimal look-back horizon. If a model's complexity is not carefully matched to the dataset size and this optimal horizon, increasing the look-back window or model size actually degrades performance, causing the model to get lost in high-dimensional temporal spaces.

Designing Smarter, Lightweight Forecasters

Rather than abandoning neural networks entirely, researchers have used these insights to design a new generation of lightweight, highly efficient architectures. These models discard heavy self-attention mechanisms in favor of optimized linear and MLP structures.

All-MLP Architectures

Instead of routing sequences through multi-head attention, models can achieve state-of-the-art results by using multi-layer perceptrons to map time and feature dimensions independently.

Hybrid and Regularized Transformers

When attention mechanisms are retained, they must be heavily adapted or regularized to prevent optimization failures. Xiaozhuang Song et al. demonstrated in Rethinking the Temporal Modeling for Time Series Forecasting with Hybrid Modeling that standard attention can degrade performance unless paired with well-tuned linear or CNN components.

To make Transformers viable, recent architectures modify how and where attention is applied:

The ongoing evolution of time series forecasting demonstrates that architectural sophistication is not a substitute for domain-specific inductive biases. By understanding why simple linear mappings succeed—namely, through direct mapping, lower risk of overfitting, and robust preservation of underlying dynamics—the field is moving away from bloated architectures toward lean, efficient, and mathematically grounded forecasting models.

Key Foundational Work

Revisiting Long-term Time Series Forecasting: An Investigation on Affine Mapping (2024) — contributed 2 concept elements to this direction.
Are Transformers Effective for Time Series Forecasting? (2023) — contributed 1 concept elements to this direction.
SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_complexity_performance_gap_815f1dd9,
  title = {model complexity-performance gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-complexity-performance-gap/rdr_815f1dd9ad86487dba4bc5997991edf2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_complexity_performance_gap_815f1dd9_v1,
  title = {model complexity-performance gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-complexity-performance-gap/rdr_815f1dd9ad86487dba4bc5997991edf2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake