TL;DR
In an era of increasingly bloated deep learning models, PureTS proves that "less is more" for time series forecasting. By using just three simple linear layers without any activation functions or complex modules, this model achieves state-of-the-art results in 80% of long-sequence prediction tasks while being significantly faster and lighter than its Transformer-based counterparts.
What Problem Does This Paper Address?
The field of time series forecasting has followed the broader trend in machine learning: moving from simple statistical models (like ARIMA) to massive, complex deep learning architectures (like Transformers and GNNs). While these complex models show incremental accuracy gains, they do so at the cost of massive parameter counts and slow inference speeds.
The authors identify a specific pathology in complex models they call "over-fluctuating." Because deep models have high capacity and non-linear activation functions, they often try to mimic every jagged peak and noise-filled valley in the training data. When predicting far into the future (long-term forecasting), this leads to distorted curves that fail to capture the underlying "core rules" or general trends of the data. The challenge is: can we build a model that ignores the noise and captures only the essential temporal laws?
Methods
The authors propose PureTS, a remarkably minimalist architecture. Its design philosophy is built on three pillars:
- Pure Linearity: The model consists of only three linear layers. There are no activation functions (like ReLU or Sigmoid), no dropout, and no normalization layers.
- The "Permute" Operation: To handle temporal data, the model transposes the input matrix so that the linear layers project across the time dimension rather than just the feature dimension. This allows the model to learn weights for specific historical time steps relative to the prediction horizon.
- PureTS_S: For multivariate datasets with high feature dimensions, they introduce a variant that includes a spatial projection layer to handle inter-variable correlations.
The authors argue that by removing non-linearity, they restrict the model's "prediction space" to a more robust, constrained set of functions that are naturally better at representing long-term trends.
Datasets
The researchers evaluated PureTS against a wide battery of standard benchmarks:
- Long-sequence datasets: ETT (Electricity Transformer Temperature - versions h1, h2, and m1), Weather, and Electricity.
- Short-sequence datasets: Exchange-Rate, Solar-Energy, and Traffic.
These datasets vary significantly in granularity (from 15-minute intervals to daily rates) and feature complexity (from 7 variables to 862).
Key Insights and Ideas
- The Occam’s Razor of Forecasting: The most surprising finding is that a linear model can outperform Transformers on tasks where Transformers were specifically designed to excel. This suggests that for many real-world time series, the "core rules" are closer to linear than we previously assumed.
- Linearity as a Regularizer: By intentionally limiting the model's capacity (removing non-linearities), the authors created a "bottleneck" that forces the model to ignore high-frequency noise and focus on the fundamental periodicities and trends.
- Avoiding the "Jagged Curve": The paper demonstrates that non-linear models produce "irregular polylines" in their predictions. PureTS produces smoother, more physically plausible curves for long-distance forecasting.
- Design Standards for Lightweight Models: The authors suggest that for long-step tasks, models should aim to keep input and output dimensions the same and avoid "fragmentation" (breaking the data into too many small sub-operations).
Key Results
- SOTA Performance: PureTS achieved state-of-the-art (SOTA) results in 80% of the long-sequence prediction tasks tested.
- Massive Efficiency Gains: In the ETTh1 dataset, PureTS achieved an MSE of 0.473 (at horizon 720), significantly better than the famous Informer model (0.941) and even the MLP-based LightTS (0.542).
- Speed and Scale: PureTS is nearly the lightest model in existence for these tasks. Figure 2 in the paper shows it has the lowest MACs (Multiply-Accumulate operations) and the fastest runtime, making it ideal for edge computing or real-time applications.
- Convergence: Linear models converge much faster during training than non-linear ones, as shown in the paper's convergence rate analysis.
Limitations and Future Work
Despite its empirical success, the authors acknowledge a few hurdles:
- The "Why" Question: While the model works, the authors admit they cannot yet theoretically explain why a purely linear model is so effective at approximating what are presumably non-linear real-world systems.
- High Feature Complexity: PureTS struggles more on datasets with extremely high feature dimensions (like Solar-Energy with 137 features), where its performance was less effective than the PureTS_S variant. This indicates that spatial correlation still requires some specialized structural design.
- Future Directions: The team plans to investigate the mathematical connection between linear projection maps and the "real curves" of complex time series to bridge the gap between their empirical results and theoretical understanding.
Summary updated