In deep learning, the dominant paradigm is clear: larger architectures with more parameters and complex attention mechanisms yield superior performance. This holds true across natural language processing and computer vision. However, long-term time series forecasting (LTSF) presents a striking anomaly. In this domain, highly sophisticated architectures frequently underperform relative to "embarrassingly simple" linear models and Multi-Layer Perceptrons (MLPs).
Understanding this complexity-performance paradox requires looking closely at the structural properties of temporal data, the pitfalls of over-parameterization, and the specific inductive biases required to map the past to the future.
The Empirical Discovery of the Complexity Gap
The tension between model complexity and forecasting accuracy came to the forefront when Ailing Zeng et al. published Are Transformers Effective for Time Series Forecasting?. The authors demonstrated that a family of single-layer linear models (LTSF-Linear) consistently outperformed complex, state-of-the-art Transformer architectures on standard benchmarks, often by margins as large as 20% to 50%.
This finding challenged the assumption that self-attention mechanisms designed for natural language are inherently suited for numerical sequences. Rather than extracting deep semantic features, the heavy lifting in successful neural forecasters is often driven by simple affine mappings. As shown by Zhe Li et al. in Revisiting Long-term Time Series Forecasting: An Investigation on Affine Mapping, these linear mappings are exceptionally proficient at capturing periodic patterns. When the weights of complex feature extractors are left unoptimized or over-engineered, their contribution to the actual forecast is minimal, raising fundamental questions about the necessity of intricate designs.
Why Complexity Fails: Information Density and Overfitting
Why do massive architectures struggle where a single linear layer excels? The answer lies primarily in the difference in information density and the risk of overfitting.
Low Information Density
Natural language tokens carry rich, discrete semantic meaning. In contrast, continuous numerical time series data has a much lower information density. When massive models are applied to these sequences, their high parameter count becomes a liability. Liangwei Nathan Zheng et al. explored this in Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis: The Impact of Modality Alignment, demonstrating that 32-layer Large Language Models often fail to outperform simple linear models because their massive capacity leads to overfitting rather than the discovery of true temporal patterns. This over-parameterization trap also impacts generative architectures; Jiaxin Gao et al. observed in Auto-Regressive Moving Diffusion Models for Time Series Forecasting that using heavy Transformer backbones as diffusion operators introduces unnecessary complexity, degrading performance on smaller datasets.
Temporal Distribution Shifts
Complex models are highly adept at memorizing the specific quirks of their training data. In time series, however, environments evolve due to shifting demographics, policy changes, or seasonal variations. Helen Zhou et al. highlighted this vulnerability in Evaluating Model Performance in Medical Datasets Over Time. While high-capacity models (such as gradient-boosted trees) perform exceptionally well on contemporary data, they exhibit lower robustness to temporal distribution shifts. As the gap between training and deployment increases, simpler linear models eventually surpass them because they do not overfit to era-specific features.
Mismatched Inductive Biases
Many time series originate from deterministic, well-understood physical or economic processes. Nga Yu Lo et al. demonstrated in When Bigger isn't Better: The Role of Model-Data Complexity in Time Series Forecasts that while massive foundation models can handle highly intricate, noisy data, they underperform on basic, deterministic time series. They fail to leverage their capacity for simple patterns where classical statistical models with appropriate inductive biases naturally thrive.
The Mechanics of Success: Direct Mapping and Learning Dynamics
The success of linear models is not accidental; it is rooted in how they preserve and map temporal structures.
A primary reason complex sequence models fail is their lack of specific structural priors required to map historical sequences directly to future values. This concept is formalized through the PRO-DYN framework by Alexis-Raja Brachet et al. in Dynamics is what you need for time-series forecasting!. The authors argue that a model's forecasting success depends heavily on having a learnable block at the very end of the architecture that directly maps past observations to future predictions.
Direct Linear Mapping (LTSF-Linear / PRO-DYN):
[Past Horizon: X_1, ..., X_t] ----(Learnable Linear Weights)----> [Future Forecast: Y_1, ..., Y_h]
Indirect Mapping (Standard Transformer):
[Past Horizon] --> [Temporal Embedding] --> [Self-Attention Layers] --> [Projection] --> [Future Forecast]
When architectures lack this direct mapping, they struggle to preserve the underlying data dynamics. In Time Series Forecasting Through the Lens of Dynamics, the authors show that simply appending a learnable linear dynamics layer to the end of underperforming, complex models can significantly boost their accuracy, proving that learning direct temporal dynamics is the missing key.
Furthermore, the relationship between the historical context window and model complexity is tightly constrained. As proven by Jingzhe Shi et al. in Scaling Law for Time Series Forecasting, scaling laws do exist in time series forecasting, but they are governed by an optimal look-back horizon. If a model's complexity is not carefully matched to the dataset size and this optimal horizon, increasing the look-back window or model size actually degrades performance, causing the model to get lost in high-dimensional temporal spaces.
Designing Smarter, Lightweight Forecasters
Rather than abandoning neural networks entirely, researchers have used these insights to design a new generation of lightweight, highly efficient architectures. These models discard heavy self-attention mechanisms in favor of optimized linear and MLP structures.
All-MLP Architectures
Instead of routing sequences through multi-head attention, models can achieve state-of-the-art results by using multi-layer perceptrons to map time and feature dimensions independently.
- TiDE: Introduced by Abhimanyu Das et al. in Long-term Forecasting with TiDE: Time-series Dense Encoder, this architecture uses an MLP-based encoder-decoder structure. It retains the computational speed of linear models while successfully handling non-linear dependencies and exogenous covariates, running 5x to 10x faster than standard Transformers.
- TSMixer: Developed by Si-An Chen et al. in TSMixer: An All-MLP Architecture for Time Series Forecast-ing, this model systematically interleaves simple mixing operations across time and feature dimensions. It proves that the heavy computational overhead of attention is often completely unnecessary when simpler temporal patterns dominate.
- ReNF: In ReNF: Rethinking the Design Space of Neural Long-Term Time Series Forecasters, Yihang Lu et al. paired a simple MLP architecture with a "Boosted Direct Output" paradigm and weight smoothing techniques, outperforming complex Transformers while operating up to 20x more efficiently.
Hybrid and Regularized Transformers
When attention mechanisms are retained, they must be heavily adapted or regularized to prevent optimization failures. Xiaozhuang Song et al. demonstrated in Rethinking the Temporal Modeling for Time Series Forecasting with Hybrid Modeling that standard attention can degrade performance unless paired with well-tuned linear or CNN components.
To make Transformers viable, recent architectures modify how and where attention is applied:
- Smoothing the Loss Landscape: Transformers applied to time series often suffer from optimization instability and sharp loss landscapes. To resolve this, Romain Ilbert et al. introduced SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. By combining a lightweight, channel-wise attention mechanism with Sharpness-Aware Minimization (SAM), they smooth out the loss landscape, allowing a tiny model to compete with massive foundation models.
- Refocusing on Cross-Variable Dependencies: In multivariate forecasting, temporal attention can be redundant. Jiaxin Gao et al. introduced the Client architecture in Client: Cross-variable linear integrated enhanced transformer for multivariate long-term time series forecasting. This model offloads trend forecasting to a dedicated linear module and shifts the Transformer's attention mechanism away from the time dimension entirely, focusing it instead on cross-variable interactions.
The ongoing evolution of time series forecasting demonstrates that architectural sophistication is not a substitute for domain-specific inductive biases. By understanding why simple linear mappings succeed—namely, through direct mapping, lower risk of overfitting, and robust preservation of underlying dynamics—the field is moving away from bloated architectures toward lean, efficient, and mathematically grounded forecasting models.
Key Foundational Work
Revisiting Long-term Time Series Forecasting: An Investigation on Affine Mapping (2024) — contributed 2 concept elements to this direction.
Are Transformers Effective for Time Series Forecasting? (2023) — contributed 1 concept elements to this direction.
SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (2024) — contributed 1 concept elements to this direction.