Introduction
Transformers, originally designed for discrete Natural Language Processing (NLP) tokens, often struggle with continuous time series data. While complex variants like Informer or ContiFormer have been proposed to handle long-term dependencies, simple linear models frequently outperform them on standard benchmarks. This research explores the minimal necessary modifications to adapt a standard Transformer for time series forecasting, aiming to solve the "dimension dilemma"—the conflict between high-dimensional positional requirements and low-dimensional model capacity.
Methodology
The study introduces two primary architectural refinements:
- MiTS-Transformer (Minimal Time Series Transformer): This model replaces the standard discrete
nn.Embeddinglayer with a simplenn.Linearlayer. This "vector-to-vector" mapping projects continuous input values into the model's internal dimension ($d_{model}$), with a corresponding "un-embedding" layer at the output. - PoTS-Transformer (Positional Encoding Expansion): To handle long sequences without overfitting, the PoTS approach decouples positional resolution from model capacity. It wraps the positional encoder between two linear layers, expanding the model vector to a higher dimension only for the positional encoding step. This allows for high-definition time awareness while keeping the core Transformer layers small and parameter-efficient.
Experimental Setup
The models were evaluated using synthetic sinusoid sequences of varying frequencies. The task was a sequence-to-sequence problem: predicting the next 12 samples from an initial 19-sample window.
Results and Discussion
The MiTS-Transformer successfully converged on fixed-frequency data, validating the linear mapping approach for continuous signals.
On more complex datasets with arbitrary frequencies, the PoTS-Transformer demonstrated superior performance. By expanding the positional dimension while keeping the core model small, it achieved better accuracy than larger MiTS models. For instance, increasing $d_{model}$ in MiTS from 8 to 128 increases parameters by 158x, whereas the PoTS approach achieves similar high-resolution awareness with only a 2.4x increase in parameters, effectively mitigating overfitting.
Conclusion
Minimalist adaptations can make Transformers highly effective for time series. The PoTS technique specifically addresses the trade-off between positional detail and model size. While synthetic data provided a clear proof-of-concept, future work is required to validate these minimal models against stochastic, real-world datasets like ETT or Electricity.
Summary updated