Minimal Time Series Transformer

Joni-Kristian KämäräinenarXiv 2025 · DOI · arXiv

Introduction

Transformers, originally designed for discrete Natural Language Processing (NLP) tokens, often struggle with continuous time series data. While complex variants like Informer or ContiFormer have been proposed to handle long-term dependencies, simple linear models frequently outperform them on standard benchmarks. This research explores the minimal necessary modifications to adapt a standard Transformer for time series forecasting, aiming to solve the "dimension dilemma"—the conflict between high-dimensional positional requirements and low-dimensional model capacity.

Methodology

The study introduces two primary architectural refinements:

  1. MiTS-Transformer (Minimal Time Series Transformer): This model replaces the standard discrete nn.Embedding layer with a simple nn.Linear layer. This "vector-to-vector" mapping projects continuous input values into the model's internal dimension ($d_{model}$), with a corresponding "un-embedding" layer at the output.
  2. PoTS-Transformer (Positional Encoding Expansion): To handle long sequences without overfitting, the PoTS approach decouples positional resolution from model capacity. It wraps the positional encoder between two linear layers, expanding the model vector to a higher dimension only for the positional encoding step. This allows for high-definition time awareness while keeping the core Transformer layers small and parameter-efficient.

Experimental Setup

The models were evaluated using synthetic sinusoid sequences of varying frequencies. The task was a sequence-to-sequence problem: predicting the next 12 samples from an initial 19-sample window.

Type 2 synthetic sinusoid data showing the 19-sample input (X) and 12-sample target (Y) for various frequencies.

Results and Discussion

The MiTS-Transformer successfully converged on fixed-frequency data, validating the linear mapping approach for continuous signals.

Comparison of model predictions after 100 and 600 epochs, showing convergence on the sinusoid forecasting task.

On more complex datasets with arbitrary frequencies, the PoTS-Transformer demonstrated superior performance. By expanding the positional dimension while keeping the core model small, it achieved better accuracy than larger MiTS models. For instance, increasing $d_{model}$ in MiTS from 8 to 128 increases parameters by 158x, whereas the PoTS approach achieves similar high-resolution awareness with only a 2.4x increase in parameters, effectively mitigating overfitting.

Conclusion

Minimalist adaptations can make Transformers highly effective for time series. The PoTS technique specifically addresses the trade-off between positional detail and model size. While synthetic data provided a clear proof-of-concept, future work is required to validate these minimal models against stochastic, real-world datasets like ETT or Electricity.

Summary updated

BibTeX
@misc{kamarainen2025minimal,
  author = {Joni-Kristian Kämäräinen},
  title = {{Minimal Time Series Transformer}},
  year = {2025},
  eprint = {2503.09791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.09791},
}
Figure