In machine learning, we often operate under the assumption that greater architectural complexity—more parameters, deeper networks, and expressive probabilistic modeling—inherently translates to superior predictive accuracy. However, across a wide variety of sequential and time-series forecasting domains, a sobering paradox has emerged: highly sophisticated deep learning architectures frequently struggle to outperform "naive" baselines.
Whether predicting retail sales, traffic speeds, or pandemic trajectories, simple heuristics—such as assuming the future will exactly mirror the most recent observation (the persistence baseline) or calculating a basic historical average—regularly match or exceed the performance of state-of-the-art neural networks. As Riccardo Ughi et al. argue in Two Steps Forward and One Behind: Rethinking Time Series Forecasting with Deep Learning, the field frequently neglects these simple non-parametric baselines, leading to the adoption of heavy architectures whose actual predictive utility is illusory. To build better models, we must first understand the mathematical and structural reasons why these trivial baselines are so difficult to beat.
The Power of System Inertia and Autocorrelation
The primary reason naive baselines are so formidable is that many real-world processes are governed by strong temporal dependence, or physical inertia. In short-horizon forecasting, the immediate past contains an overwhelming amount of mutual information about the immediate future.
Consider short-term wind forecasting. Chris Barber et al. demonstrated in Auto-Regressive HMM Inference with Incomplete Data for Short-Horizon Wind Forecasting that simple persistence models often outperform complex physical simulations because capturing the immediate inertia of the weather system is paramount for short-range accuracy. Similarly, in building control systems, Qie Hu et al. found in Building Model Identification during Regular Operation - Empirical Results and Challenges that a persistence-based heuristic assuming internal heat gains remain constant over short intervals provides a remarkably robust baseline for online learning without requiring complex real-time sensing.
When an environment is highly autocorrelated, a zero-intelligence model that simply repeats the last observation acts as an incredibly accurate anchor. Albert Bifet et al. highlighted this in Evaluation methods and decision theory for classification of streaming data with temporal dependence, showing that without benchmarking against a naive persistent baseline, researchers can easily misattribute a model's success to its complexity when it is actually underperforming compared to simple repetition. This phenomenon spans stable environments like optical networks (Somaiieh Yousefi et al. in Forecasting Lightpath Quality of Transmission and Implementing Uncertainty in the Forecast Models) and highly volatile ones like short-term financial forecasting (Aleksandr Pletnev et al. in Graph Neural Networks for Model Recommendation using Time Series Data), where persistence remains the optimal choice in up to 43% of cases.
Overfitting to Noise vs. Modeling Signal
If system inertia explains why simple baselines are strong, what explains why complex models are weak? The answer lies in the trade-off between variance and bias. High-capacity models like deep LSTMs, Transformers, and complex probabilistic architectures are designed to capture intricate, non-linear relationships. However, real-world time-series data is often dominated by transient noise, structural shifts, or data sparsity.
When evaluated on point estimation metrics like RMSE, sophisticated probabilistic models often degrade because the complexity required to map full probability distributions introduces noise and leads to overfitting. In a large-scale analysis of sales forecasting, Seungjae Jung et al. found in A Worrying Analysis of Probabilistic Time-series Models for Sales Forecasting that simple multi-layer perceptrons and linear regressions consistently outperformed complex autoregressive or state-space models. The authors noted that architectural sophistication does not inherently solve the practical challenges of noisy retail environments.
This pattern repeats across diverse domains:
- Energy and Traffic: In electrical load and traffic forecasting, simpler models like feed-forward networks, tree ensembles, or even basic physical heuristics frequently outperform deep recurrent networks (Zhixian Wang et al. in Benchmarks and Custom Package for Electrical Load Forecasting; Thomas Epelbaum et al. in Deep Learning applied to Road Traffic Speed forecasting; Xinyue Cai et al. in Traffic Flow Prediction: A Method Using Bagging-Based Ensemble Learning Model).
- Epidemiology and Crisis Tracking: During rapid shifts in disease dynamics or humanitarian crises, complex models struggle to capture real-time signals amidst digital behavioral noise or volatile environments. Simple baselines projecting flat trends or historical averages often perform comparably or superiorly (Shihao Yang et al. in Use Internet search data to accurately track state level influenza epidemics; Velma K. Lopez et al. in Challenges of COVID-19 Case Forecasting in the US, 2020-2021; Govind Jeevan et al. in Accuracy of US CDC COVID-19 forecasting models; Rahul Nair et al. in An Explainable Forecasting System for Humanitarian Needs Assessment).
- Financial NLP: Even when applying Large Language Models to earnings transcripts, Ding Yu et al. showed in Same Company, Same Signal: The Role of Identity in Earnings Call Transcripts that a dead-simple baseline using only a company's historical volatility outperforms state-of-the-art LLMs. The complex models merely approximate the long-term statistical distribution rather than extracting novel semantic signals from the text.
Historical Averages as Robust Regularizers
When a system moves beyond immediate short-term horizons where persistence dominates, the next most effective baseline is often a historical average or frequency metric. While theory-based computational models or deep architectures attempt to simulate underlying cognitive or physical mechanisms, basic historical frequency acts as a powerful regularizer.
For instance, in behavior prediction, Chao Zhang et al. found in Theory-based habit modeling for enhancing behavior prediction in behavior change support systems that complex psychological models failed to provide immediate predictive gains over simple past-behavior averages. Similarly, for dynamic demand response in smart buildings, Saima Aman et al. demonstrated in Prediction Models for Dynamic Demand Response that averaging models using only a few days of recent data offered the optimal balance of accuracy and computational efficiency. By compressing historical patterns into a single mean or frequency, these models eliminate the variance that causes deeper architectures to fail in univariate or noisy contexts (Mingli Song et al. in A robust federated biased learning algorithm for time series forecasting).
Restoring Rigor to Time-Series Evaluation
The recurring failure of complex models to decisively beat naive baselines highlights a systemic issue in machine learning research: the lack of rigorous benchmarking. To truly validate a new architecture, it must be tested against non-parametric baselines that exploit system inertia and historical distributions.
If a deep Transformer model (Shihao Yang et al. in County augmented transformer for COVID-19 state hospitalizations prediction) or a massive multi-model ensemble (Mostafa Farrokhabadi et al. in Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm) cannot outperform a flat-line projection or a historical average, its computational cost and deployment complexity are unjustified. Understanding the strength of these simple baselines allows researchers to design architectures that intentionally model what the baseline misses, rather than accidentally reinventing a worse version of a simple average.
Key Foundational Work
Auto-Regressive HMM Inference with Incomplete Data for Short-Horizon Wind Forecasting (2010) — contributed 1 concept elements to this direction.
Experiments in UNIX Command Prediction (1997) — contributed 1 concept elements to this direction.