Hybrid Architectures for Local and Global Sequence Modeling

This direction is split into other directions: multi-scale temporal modeling (local-global dependency trade-off). Its last version stays readable and citable here.

Version 1 of 188 papers92 conceptsfrozen 2026-10-04

Time series data is rarely a single signal; it is a composite of two distinct temporal scales. On one hand, there are local fluctuations: the sudden spike in a stock price, a facial micro-expression, or a transient heartbeat anomaly. On the other, there are global trends: the quarterly earnings cycle, the gradual shift in a patient’s mood, or the seasonal rhythm of city traffic.

The central challenge in modern sequence modeling is that architectures optimized for one scale often fail at the other. Convolutional Neural Networks (CNNs) are excellent at extracting fine-grained local patterns but have a limited "receptive field," making them blind to long-term context. Conversely, Transformers excel at capturing global dependencies through self-attention but can overlook subtle, point-to-point deviations or become computationally prohibitive when processing high-resolution local data.

To bridge this gap, researchers are converging on hybrid architectures that explicitly separate and then fuse local and global feature extraction.

The Convolutional Tokenizer

One of the most intuitive ways to balance these scales is to use a "local-first" approach. Instead of feeding raw, high-frequency data directly into a global model, researchers use a local feature extractor to "tokenize" the sequence.

In the CTTS framework, Zhen Zeng and colleagues use CNNs to identify immediate price movements in financial data. These local features are then passed to a Transformer, which models the broader market trend. Similarly, for medical diagnostics, Safwen Naimi developed a 1D-Convolutional transformer that captures fine-grained gait patterns via convolutions while maintaining the long-term spatio-temporal relationships necessary to diagnose Parkinson’s disease.

By using convolutions as a front-end, these models reduce the sequence length for the Transformer, allowing it to focus on "global" reasoning without being overwhelmed by "local" noise.

Parallel and Hierarchical Fusion

Rather than a sequential pipeline, some architectures process both scales in parallel. The Time-Transformer, proposed by Yuansan Liu and team, runs a Temporal Convolutional Network (TCN) alongside a Transformer. A bidirectional cross-attention mechanism allows these two branches to "talk" to each other, ensuring that the global trend informs the local reconstruction and vice versa.

Hierarchical approaches take this further by progressively merging segments of time. Zezhi Shao introduced HUTFormer, which uses a U-Net inspired structure. It employs windowed attention for local details and segment merging to capture periodic cycles. This hierarchy ensures that the model doesn't have to choose between resolution and range; it maintains both by looking at the data through lenses of varying magnification.

Beyond the Time Domain

Sometimes, the "global" view is best captured not by looking further back in time, but by looking at the data differently. TLNets, developed by Yang Liu and Hao Sun, use mathematical transformations like Fourier Transforms and SVD to achieve a global receptive field instantaneously. By fusing these frequency-domain views with local convolutional features, the model identifies periodicities that a standard time-domain model might miss.

This frequency-enhanced perspective is also critical for anomaly detection. In FCVAE, Changhua Pei demonstrates that using global frequency features as "conditions" for a Variational Autoencoder allows the model to reconstruct normal rhythms more accurately, making subtle, local deviations (anomalies) stand out more clearly.

The Rise of State-Space Hybrids

The most recent evolution in this field involves State-Space Models (SSMs) like Mamba, which offer linear-time efficiency. While SSMs are excellent at maintaining a "running summary" of the past, they can struggle with the precise, high-resolution recall that Transformers provide.

New hybrid models, such as those described by Kexin Tang in cryptocurrency prediction, combine Mamba’s efficiency for local patterns with Transformer attention for global context. Similarly, WaveSSM by Ruben Solozabal replaces global polynomial bases with localized wavelet frames. This allows the model to "attend" to transient, non-stationary events—like a sudden heartbeat spike—while still scaling to the extreme long-range dependencies of the signal.

Fast and Slow Reasoning

This research direction mirrors human cognition. As framed by Yuting Wu in the CognTKE framework, effective prediction requires both "fast" global shallow reasoning (recognizing a familiar historical pattern) and "slow" local deep reasoning (analyzing a complex, recent chain of events).

By designing architectures that respect this duality, researchers are moving away from "one-size-fits-all" layers and toward systems that can simultaneously see the forest and the trees. Whether through CNN-Transformer hybrids, frequency-domain integration, or wavelet-based state spaces, the goal remains the same: ensuring that the model's awareness of the distant past never comes at the cost of its sensitivity to the immediate present.

Papers in this version

Showing 60 of 88 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_architectures_for_local_and_globa_5792631f,
  title = {Hybrid Architectures for Local and Global Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-architectures-for-local-and-global-sequence-modeling/rdr_5792631f2f154053b07e1e8421d7c981},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_architectures_for_local_and_globa_5792631f_v1,
  title = {Hybrid Architectures for Local and Global Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-architectures-for-local-and-global-sequence-modeling/rdr_5792631f2f154053b07e1e8421d7c981/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}