Temporal Order and Permutation Invariance in Time Series

This direction is split into other directions: temporal order preservation (permutation invariance). Its last version stays readable and citable here.

Version 1 of 1111 papers129 conceptsfrozen 2026-10-04

In a surprising "Emperor’s New Clothes" moment for deep learning, Ailing Zeng, Muxi Chen, and colleagues published Are Transformers Effective for Time Series Forecasting?, demonstrating that simple one-layer linear models consistently outperform sophisticated Transformer architectures. This finding exposed a fundamental mismatch between the core mechanism of the Transformer—self-attention—and the nature of time series data.

The central issue is permutation invariance. While Transformers treat sequences as unordered sets of tokens, time series data is defined entirely by its chronological order.

The Permutation Invariance Problem

To understand why this matters, consider the difference between a sentence and a temperature reading. In natural language, if you shuffle the words in a sentence, a human (and a Transformer) can often still infer the semantic meaning because words have inherent "content." In a time series, if you shuffle the data points, the "content" (the numerical value) remains, but the "context" (the trend, seasonality, and momentum) is completely destroyed.

Standard self-attention is mathematically invariant to the order of its inputs. As Qi Liu and Zhi Li argue in FormerTime, this mechanism treats relationships between time steps based on semantic correlation rather than temporal proximity. If you shuffle the input sequence, a Transformer's output remains largely stable, whereas a linear model’s performance collapses. This suggests that Transformers often ignore the very thing that makes a time series a series: the sequence.

Why Positional Encodings Aren't Enough

The standard fix for permutation invariance is adding Positional Encodings (PE). However, research suggests these are often insufficient for numerical sequences. Seyed Navid Mohammadi Foumani and Chang Wei Tan point out in ConvTran that standard PEs struggle to capture the continuous temporal shape or long-term dependencies within a lookback window.

When data is irregular—common in medical or financial settings—the problem intensifies. Byunghyun Kim and Jae-Gil Lee note in Continuous-Time Linear Positional Embedding that Transformers cannot distinguish the timing of data points without explicit, often complex, structural modifications. Without a strong temporal inductive bias, the model treats the sequence as a "bag of observations" rather than a physical process unfolding over time.

Inverting the Architecture

If the problem is that Transformers look at time steps as tokens, one solution is to flip the logic entirely. In iTransformer, Haixu Wu and Tengge Hu propose "inverting" the Transformer. Instead of treating each time step as a token, they treat the entire history of a single variable (a "variate") as a token.

By applying attention across variates rather than across time, the model uses the attention mechanism to capture multivariate correlations while using simple feed-forward networks to handle the temporal sequence. This sidesteps the permutation invariance issue by letting the MLP—which is naturally sensitive to order—process the time dimension.

Beyond Attention: Order-Aware Alternatives

The realization that self-attention may be "too flexible" for time series has led to a resurgence of architectures with built-in order awareness:

  • State Space Models (SSMs): Models like Mamba, explored by Jindong Han, use recurrent-like structures that naturally respect temporal order while maintaining the parallelizable training of Transformers.
  • Linear and MLP Models: LTSF-Linear and MLP4Rec by Muyang Li use weighted sums of historical values, which are inherently dependent on the specific sequence of values.
  • Convolutional Hybrids: MSDCN by Suhan Guo uses dilated convolutions to capture multi-scale patterns, leveraging the local temporal smoothness that permutation-invariant models often miss.

The research direction is clear: for time series, "more complex" is not always "better." The field is moving away from generic sequence models toward architectures that explicitly incorporate the chronological constraints of the physical world. As Jiecheng Lu suggests in the In-context Time Series Predictor, the future of the field lies in aligning the high-capacity learning of Transformers with the strict, sequential reality of time.

Papers in this version

Showing 60 of 111 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_order_and_permutation_invarianc_8d309c4b,
  title = {Temporal Order and Permutation Invariance in Time Series},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-order-and-permutation-invariance-in-time-series/rdr_8d309c4b650b4cad9165cfbd457462ea},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_order_and_permutation_invarianc_8d309c4b_v1,
  title = {Temporal Order and Permutation Invariance in Time Series},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-order-and-permutation-invariance-in-time-series/rdr_8d309c4b650b4cad9165cfbd457462ea/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}