Linear Architectures for Long-Term Time Series Forecasting

This direction is split into other directions: model complexity-performance gap. Its last version stays readable and citable here.

Version 1 of 190 papers100 conceptsfrozen 2026-10-04

In the landscape of deep learning, the Transformer is often viewed as a universal architecture. Its success in natural language processing and computer vision led to an era where "Attention is All You Need" became the default starting point for almost any sequential task. However, in the specific domain of long-term time series forecasting (LTSF), this dominance has been challenged by a surprising "Emperor’s New Clothes" moment: simple linear models frequently outperform complex, multi-million parameter Transformers.

The Complexity Paradox

The central tension in recent research is that sophisticated attention mechanisms often introduce more noise than predictive power when applied to numerical sequences. Ailing Zeng and colleagues famously demonstrated in Are Transformers Effective for Time Series Forecasting? that a set of "embarrassingly simple" one-layer linear models, dubbed LTSF-Linear, consistently beat state-of-the-art Transformers across major benchmarks.

This performance gap suggests that the expressive power of Transformers is not being effectively utilized. As Yang Ke and others argue in Curse of Attention, large-scale attention models fail to generalize on time series even when over-parameterized and trained on extensive datasets, presenting a paradox for standard scaling laws.

Why Standard Attention Fails Time Series

The failure of standard Transformers in this domain stems from a fundamental mismatch between the architecture and the data structure:

  1. Permutation Invariance: Self-attention is inherently permutation-invariant; it treats a sequence as a "bag of tokens" and relies on positional encodings to regain order. In contrast, time series data is defined by its temporal order. Ailing Zeng argues that for numerical sequences, a basic weighted sum of historical values (a linear mapping) is more effective at capturing these dependencies than the complex relational modeling of attention.
  2. Structural Limitations: Theoretical work by Yufa Zhou in Why Do Transformers Fail to Forecast Time Series In-Context? proves that linear self-attention is mathematically restricted to a "compressed" version of linear regression. This creates a structural performance gap that only vanishes as context length approaches infinity.
  3. Sharp Loss Landscapes: Transformers in TSF often suffer from optimization instability. Romain Ilbert and Ambroise Odonnat identify that these models often converge to "sharp" minima, which generalize poorly to the distribution shifts common in real-world temporal data.

The Back to Basics Movement

In response, a new generation of models has emerged that prioritizes linear mappings and Multi-Layer Perceptrons (MLPs). These models, such as TSMixer by Si-An Chen and TiDE by Abhimanyu Das, replace attention layers with simple operations that "mix" information across time and features.

The intuition behind these models is that time series forecasting is often a problem of capturing periodic trends and global patterns, which linear layers do naturally. For instance, Zhe Li shows in Revisiting Long-term Time Series Forecasting that linear mappings are mathematically optimized to capture the periodic nature of time series data, especially when paired with techniques like reversible normalization (RevIN).

Inverting the Transformer

Rather than abandoning the Transformer entirely, some researchers have sought to "fix" its orientation. In iTransformer, Tengge Hu, Haixu Wu, and colleagues propose inverting the architecture.

In a standard Transformer, tokens represent time steps, and attention looks for correlations between different points in time. In an iTransformer, tokens represent entire "channels" (variables), and attention looks for correlations between different variables. This allows the model to use its attention mechanism for multivariate correlation while using simple feed-forward networks to handle the temporal dimension, effectively combining the strengths of both paradigms.

Smoothing the Path to Generalization

Another branch of research focuses on the optimization of these models. SAMformer, introduced by Romain Ilbert, combines a lightweight channel-wise attention architecture with Sharpness-Aware Minimization (SAM). By explicitly seeking "flatter" regions of the loss landscape during training, SAMformer overcomes the optimization hurdles that typically cause Transformers to underperform relative to linear baselines.

The shift in the field is clear: the goal is no longer just to increase model capacity, but to align the architecture's inductive biases with the fundamental properties of time—order, periodicity, and noise. Whether through purely linear models like PureTS or hybrid approaches like iTransformer, the research direction is moving toward structural simplicity and optimization robustness.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_architectures_for_long_term_time__8a604165,
  title = {Linear Architectures for Long-Term Time Series Forecasting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-architectures-for-long-term-time-series-forecasting/rdr_8a604165b45c4b8a9bc839d8121e083c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_architectures_for_long_term_time__8a604165_v1,
  title = {Linear Architectures for Long-Term Time Series Forecasting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-architectures-for-long-term-time-series-forecasting/rdr_8a604165b45c4b8a9bc839d8121e083c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}