Temporal Patching for Time Series Transformers

315 concepts232 papers1 proposal

To understand why temporal patching has revolutionized time-series analysis, we must first look at why traditional deep learning architectures struggled with sequential numerical data. Historically, models processed time series point by point—treating each individual timestamp's value as a single token.

This approach suffers from two fundamental flaws: a lack of semantic meaning and an existential computational bottleneck. In natural language processing, a token is a word or sub-word carrying rich semantic context. In contrast, a single numerical data point (such as a single temperature reading or stock price) contains virtually no semantic information on its own and is highly susceptible to local noise. Furthermore, the self-attention mechanism in Transformers scales quadratically ($O(L^2)$) with the sequence length $L$. For long-term forecasting, where models need to look back across thousands of historical time steps, point-wise tokenization makes the sequence length prohibitively large, leading to massive memory and computational overhead.

Shifting from Points to Patches

Temporal patching solves both problems simultaneously by changing how models "see" continuous signals. Instead of tokenizing individual time steps, the input sequence is partitioned into contiguous, localized sub-series segments called patches.

As introduced by Yuqi Nie et al. in A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, these patches are mapped into a high-dimensional embedding space via a linear projection or a 1D convolution. This structural shift provides two immediate benefits:

  1. Quadratic Efficiency Gains: If a time series of length $L$ is divided into patches of length $P$, the effective sequence length drops to $L/P$. Because attention complexity is quadratic, the computational overhead shrinks to $O((L/P)^2)$, allowing models to ingest significantly longer historical windows.
  2. Semantic Preservation: A patch naturally captures local temporal properties—such as shapes, trends, or cycles—that a single point cannot. By shifting the atomic unit of processing from a noisy individual value to a meaningful local trajectory, the model can focus on broader structural patterns.

This representational power is so potent that the underlying architecture becomes secondary. Zeying Gong et al. demonstrated in PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting that applying depthwise separable convolutions to patched data can outperform complex attention-based systems, proving that the success of modern time-series models is heavily driven by patch-based data representation itself.

Handling Multivariate Complexity

Real-world time series are rarely univariate; they often involve hundreds of interacting variables. Applying patching to multivariate data requires balancing temporal context with cross-variable dependencies.

To address this, Yunhao Zhang et al. developed Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting, which partitions time series data into local segments for each variable independently before embedding them into a two-dimensional array. This approach preserves the distinct identity and dynamics of individual dimensions while still leveraging the noise-reduction and efficiency benefits of temporal patching.

Unlocking Time-Series Foundation Models

By transforming continuous numerical sequences into discrete, token-like patches, researchers have bridged the gap between time series and natural language processing. This has unlocked the ability to build large-scale time-series foundation models and even "reprogram" existing Large Language Models (LLMs).

Tian Zhou et al. showed in One Fits All: Power General Time Series Analysis by Pretrained LM that a frozen, pre-trained language model can act as a universal computation engine for time series when patches are treated directly as tokens. Similarly, Ming Jin et al. introduced Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, which explicitly reprograms time-series patches into text-like representations to exploit the sequential reasoning capabilities of LLMs without altering their core weights.

For native time-series foundation models, patching serves as the standard input pipeline. Mononito Goswami developed MOMENT: A Family of Open Time-series Foundation Models, an open family of foundation models that utilizes non-overlapping, fixed-length patches to perform zero-shot forecasting, classification, and anomaly detection. To make this tokenization even more dynamic, Xu Liu et al. introduced Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts, which processes multi-step patches using a Sparse Mixture of Experts to autoregressively predict future patches based on historical context.

Ubiquitous Impact Across Domains

The transition to patch-based architectures has triggered widespread performance breakthroughs across specialized domains that rely on high-frequency or long-duration signals:

Ultimately, temporal patching works because it aligns the representation of numerical data with the structural strengths of modern sequence models, turning a high-frequency, noisy sequence into a clear, manageable story of local trends.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 230 papers
  2. Version 1 · 2026-10-04 · 330 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_patching_d55e8d03,
  title = {temporal patching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-patching/rdr_d55e8d0379c34013b1d9d557c4dd8f26},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_temporal_patching_d55e8d03_v2,
  title = {temporal patching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-patching/rdr_d55e8d0379c34013b1d9d557c4dd8f26/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake