PETformer: Long-term Time Series Forecasting via Placeholder-enhanced Transformer

Shengsheng Lin, Weiwei Lin, Wentai Wu, Songbo Wang, Yongxiang Wang

Introduction

The "Transformer revolution" in Long-term Time Series Forecasting (LTSF) recently faced a significant challenge. In 2023, simple linear models like Dlinear began outperforming complex Transformers, raising questions about whether the attention mechanism was suitable for temporal data. PETformer addresses this by revitalizing the Transformer architecture, achieving state-of-the-art accuracy while drastically reducing parameter overhead.

The researchers identified three primary obstacles for Transformers in LTSF:

  1. Temporal Continuity: Difficulty in maintaining the flow between historical data and future predictions.
  2. Information Density: Time points are often "low density," making it hard for models to extract meaningful patterns without complex patching.
  3. Multi-channel Relationships: The challenge of balancing channel independence with variable interaction in multivariate datasets.

The PETformer Architecture

PETformer introduces the Placeholder-enhanced Technique (PET) to overcome these hurdles. Unlike traditional models that use heavy "Flattening" layers or complex decoders, PETformer appends "learnable placeholders" directly to the end of the input sequence.

Comparison of different prediction head architectures in LTSF models.

As shown in the figure above, PETformer (d) avoids the parameter-heavy flattening layers (b) or the separate decoder structures (a). By processing history and future placeholders together in a single encoder, the attention mechanism can directly model the relationship between past events and future outcomes.

Technical Innovations

The architecture is built upon three core pillars:

  • Placeholder-enhanced Technique (PET): Future time steps are represented by learnable tokens. This allows the model to treat the forecasting task as a sequence completion problem within a unified temporal plane.
  • Token-Wise Predictor: Since placeholders are integrated into the sequence, a tiny, parameter-sharing linear layer converts each placeholder into a specific prediction. This reduces the prediction head's parameter count by over 99% compared to models like PatchTST.
  • Long Sub-sequence Division (LSD) & Multi-channel Separation and Interaction (MSI): LSD uses larger patches to capture richer semantic information, while MSI provides a flexible module for managing dependencies between different variables in multivariate data.
The internal structure of the PETformer layer, showing the integration of learnable placeholders.

The detailed layer structure illustrates how input tokens and learnable placeholders are concatenated and passed through the Transformer encoder layers, ensuring that the model learns the transition from history to future seamlessly.

Experimental Results

PETformer was evaluated against eight major public LTSF benchmarks, including the ETT datasets, Electricity, Traffic, Weather, and ILI.

Performance and Efficiency

  • State-of-the-Art Accuracy: PETformer outperformed all baselines, including PatchTST and Dlinear. It achieved a 4.7% improvement in Mean Squared Error (MSE) over PatchTST and a 24.3% improvement over Dlinear.
  • Parameter Efficiency: In a head-to-head comparison on the ETTm1 dataset, PETformer’s prediction layer required only 6.19k parameters, whereas PatchTST’s flattening layer required 8.30 million.
  • Computational Gains: The model demonstrated faster training times and significantly lower GPU memory usage, making it more practical for large-scale deployments.

Conclusion

PETformer proves that the Transformer's attention mechanism is not a liability for time series forecasting when paired with the right architectural choices. By replacing bloated prediction heads with learnable placeholders, the model achieves superior performance with a fraction of the parameters. Future research will continue to explore optimal ways to model inter-channel dependencies and scale the placeholder technique for even longer look-back windows.

Summary updated

BibTeX
@article{lin2025petformer,
  author = {Shengsheng Lin and Weiwei Lin and Wentai Wu and Songbo Wang and Yongxiang Wang},
  title = {{PETformer: Long-term Time Series Forecasting via Placeholder-enhanced Transformer}},
  journal = {IEEE Trans. Emerg. Top. Comput. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=GvjmKguC6F},
}
Figure