Convolutional Transformers for Local and Global Signal Modeling

This direction is split into other directions: hybrid convolutional-transformer architecture (local-global temporal modeling). Its last version stays readable and citable here.

Version 1 of 1130 papers142 conceptsfrozen 2026-10-04

Standard Transformer architectures, while revolutionary for natural language, often struggle when applied directly to raw, high-frequency time-series data like EEG, ECG, or seismic waves. The core issue is a mismatch in scale: self-attention is excellent at finding long-range dependencies but lacks the "inductive bias" to recognize the specific local shapes—spikes, flickers, or oscillations—that define physiological and physical signals.

To bridge this gap, a dominant research direction has emerged: the Hybrid CNN-Transformer. By integrating convolutional layers with self-attention mechanisms, researchers are creating models that can simultaneously "see" the fine-grained morphology of a single heartbeat and the hour-long rhythms of a sleep cycle.

The Blind Spot of Pure Attention

In a pure Transformer, the input is typically divided into patches or tokens. For a high-frequency signal, these tokens are often arbitrary slices of time. As Seyed Navid Mohammadi Foumani and colleagues argue in Improving Position Encoding of Transformers for Multivariate Time Series Classification, this approach ignores the local temporal context. A Transformer might know that a signal increased at minute 1 and minute 10, but it struggles to characterize the specific "n-gram-like" patterns—the sharp rise and slow decay of a seizure, for instance—that occur within those minutes.

Furthermore, raw signals are incredibly noisy. Pure attention mechanisms can be "distracted" by high-frequency jitter. Convolutions, by contrast, act as natural filters.

The Hybrid Solution: Microscope and Telescope

The most common framing of this research, seen in work by Wei Yan Peh and Yuanyuan Yao, treats the CNN as a "microscope" and the Transformer as a "telescope."

  1. The Convolutional Front-End: 1D convolutional layers process the raw waveform first. These layers have a small receptive field, allowing them to extract local morphological features—like the QRS complex in an ECG or a saccade in eye-tracking data. In DETRtime, Lukas Wolf uses this backbone to turn raw brain activity into discrete "objects" (blinks or fixations) that a Transformer can then reason about.
  2. The Transformer Backbone: Once the CNN has compressed the raw signal into a sequence of high-level feature vectors, the Transformer takes over. Because the sequence length has been reduced by the CNN's pooling or stride, the Transformer can efficiently model dependencies across much longer time horizons.

This hierarchical approach is central to SleepVST, where Jonathan F. Carter uses a shallow convolutional patch encoder to capture immediate physiological changes before a Transformer models the cyclical patterns of multi-hour sleep stages.

Interleaving for Multi-Scale Dynamics

Beyond simple front-ends, researchers are increasingly interleaving these operations. In Astroconformer, Jia-Shu Pan and Yuan-Sen Ting interleave convolutional layers directly with multi-head self-attention. This allows the model to refine local "flickers" in stellar light curves while simultaneously updating its global understanding of the star's surface gravity.

Similarly, Yiling Zhang and Dongjie Wang proposed the Multi-scale Progressive Gated Transformer, which uses temporal convolutions inside the Transformer encoder to handle "irregular local vibrations." This design recognizes that in noisy environments, local and global information must be fused progressively rather than just sequentially.

Why the Hybrid Approach Wins

The evidence across these 111 papers suggests three reasons why this hybrid direction is becoming the standard for signal processing:

  • Dimensionality Reduction: Raw sensor data is often too long for the $O(n^2)$ complexity of standard attention. CNNs act as a learned "compressor," as demonstrated by Mike A. Merrill in predicting respiratory illnesses from wearables.
  • Robustness to Noise: By using convolutional kernels, models like DiffMDD by Sha Zhao can "noise-proof" their classifications, focusing on robust signal shapes rather than point-by-point fluctuations.
  • Generalization: Hybrid models appear to generalize better across different subjects and hardware. Wei Yan Peh showed that this architecture enables patient-independent seizure detection across multiple clinical centers, a feat that pure CNNs or pure Transformers often struggle to achieve.

By combining the local precision of convolutions with the global perspective of Transformers, this research direction provides a unified framework for decoding the complex, multi-scale languages of the physical world.

Papers in this version

Showing 60 of 130 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_convolutional_transformers_for_local_and_bc6e996f,
  title = {Convolutional Transformers for Local and Global Signal Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/convolutional-transformers-for-local-and-global-signal-modeling/rdr_bc6e996f0d50421c84fc163ee20c8f4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_convolutional_transformers_for_local_and_bc6e996f_v1,
  title = {Convolutional Transformers for Local and Global Signal Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/convolutional-transformers-for-local-and-global-signal-modeling/rdr_bc6e996f0d50421c84fc163ee20c8f4c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}