Introduction
The "Transformer revolution" in Long-term Time Series Forecasting (LTSF) recently faced a significant challenge. In 2023, simple linear models like Dlinear began outperforming complex Transformers, raising questions about whether the attention mechanism was suitable for temporal data. PETformer addresses this by revitalizing the Transformer architecture, achieving state-of-the-art accuracy while drastically reducing parameter overhead.
The researchers identified three primary obstacles for Transformers in LTSF:
- Temporal Continuity: Difficulty in maintaining the flow between historical data and future predictions.
- Information Density: Time points are often "low density," making it hard for models to extract meaningful patterns without complex patching.
- Multi-channel Relationships: The challenge of balancing channel independence with variable interaction in multivariate datasets.
The PETformer Architecture
PETformer introduces the Placeholder-enhanced Technique (PET) to overcome these hurdles. Unlike traditional models that use heavy "Flattening" layers or complex decoders, PETformer appends "learnable placeholders" directly to the end of the input sequence.
As shown in the figure above, PETformer (d) avoids the parameter-heavy flattening layers (b) or the separate decoder structures (a). By processing history and future placeholders together in a single encoder, the attention mechanism can directly model the relationship between past events and future outcomes.
Technical Innovations
The architecture is built upon three core pillars:
- Placeholder-enhanced Technique (PET): Future time steps are represented by learnable tokens. This allows the model to treat the forecasting task as a sequence completion problem within a unified temporal plane.
- Token-Wise Predictor: Since placeholders are integrated into the sequence, a tiny, parameter-sharing linear layer converts each placeholder into a specific prediction. This reduces the prediction head's parameter count by over 99% compared to models like PatchTST.
- Long Sub-sequence Division (LSD) & Multi-channel Separation and Interaction (MSI): LSD uses larger patches to capture richer semantic information, while MSI provides a flexible module for managing dependencies between different variables in multivariate data.
The detailed layer structure illustrates how input tokens and learnable placeholders are concatenated and passed through the Transformer encoder layers, ensuring that the model learns the transition from history to future seamlessly.
Experimental Results
PETformer was evaluated against eight major public LTSF benchmarks, including the ETT datasets, Electricity, Traffic, Weather, and ILI.
Performance and Efficiency
- State-of-the-Art Accuracy: PETformer outperformed all baselines, including PatchTST and Dlinear. It achieved a 4.7% improvement in Mean Squared Error (MSE) over PatchTST and a 24.3% improvement over Dlinear.
- Parameter Efficiency: In a head-to-head comparison on the ETTm1 dataset, PETformer’s prediction layer required only 6.19k parameters, whereas PatchTST’s flattening layer required 8.30 million.
- Computational Gains: The model demonstrated faster training times and significantly lower GPU memory usage, making it more practical for large-scale deployments.
Conclusion
PETformer proves that the Transformer's attention mechanism is not a liability for time series forecasting when paired with the right architectural choices. By replacing bloated prediction heads with learnable placeholders, the model achieves superior performance with a fraction of the parameters. Future research will continue to explore optimal ways to model inter-channel dependencies and scale the placeholder technique for even longer look-back windows.
Summary updated