Linear Complexity Multimodal Fusion with State Space Models

This direction is split into other directions: linear-complexity multimodal fusion. Its last version stays readable and citable here.

Version 1 of 188 papers116 conceptsfrozen 2026-10-04

In the quest for robust machine perception, researchers have long sought to combine disparate sensors—RGB cameras, thermal imagers, LiDAR, and event streams. For years, the Transformer architecture was the tool of choice for this "multimodal fusion" due to its ability to model global dependencies. However, Transformers suffer from a quadratic bottleneck: as you add more sensors or increase image resolution, the computational cost explodes.

A new wave of research is replacing the Transformer's self-attention mechanism with State Space Models (SSMs), specifically the Mamba architecture. By treating multimodal fusion as a sequential state-evolution problem rather than a dense pairwise comparison, these models achieve linear scaling, enabling real-time performance on hardware where Transformers previously hit a "memory wall."

The Quadratic Bottleneck of Multimodal Attention

To understand why this shift is happening, consider the task of fusing an RGB frame with a thermal image. In a standard Transformer, every pixel in the RGB image must "attend" to every pixel in the thermal image. If each image has $N$ patches, the complexity is $O((2N)^2)$. This quadratic growth makes high-resolution or long-temporal sequences (like video tracking) prohibitively expensive.

State Space Models, popularized by the Mamba architecture, offer a different path. Instead of comparing every element to every other element, an SSM maintains a compressed "hidden state" that evolves as it scans through a sequence. This results in $O(N)$ linear complexity. Researchers like Zifu Wan and Yuhao Wang in Sigma demonstrated that this mechanism can be adapted for semantic segmentation, providing a global receptive field without the quadratic overhead.

How Modalities "Talk" Through States

The core innovation in this field is how to make different modalities interact within the linear-time scanning process. Several distinct strategies have emerged:

1. The Shared Hidden Workspace In architectures like Fusion-Mamba by Wenhao Dong, multimodal features are mapped into a shared hidden state space. As the model scans the data, the hidden state acts as a "memory" that accumulates information from both sensors. This allows the model to resolve inconsistencies—such as a pedestrian hidden in shadows in the RGB feed but visible in the infrared feed—by updating the state with the most relevant information.

2. Coupled State Chains Rather than a single scan, Wenbing Li and Hang Zhou introduced Coupled Mamba. This approach runs parallel state chains for different modalities (e.g., audio, video, and text) and "couples" them during the transition step. By summing or gating the hidden states before the next time step, the model captures inter-modality dynamics while keeping the transition matrix computationally tractable.

3. Cross-Modal Selective Scanning In tracking tasks, where a "template" of an object must be found in a "search" region, MamTrack by Chuanyu Sun and Jiqing Zhang uses bidirectional SSMs. The model scans the search region relative to the template, effectively replacing cross-attention with a selective scan that filters out background noise and focuses on the target's temporal evolution.

Efficiency in the Real World

The primary motivation for this research is deployment on edge devices and real-time systems. The efficiency gains reported across these papers are significant:

  • Autonomous Driving: DRAMA by Marcelo H. Ang Jr. replaces Transformers in motion planning, achieving higher accuracy with fewer parameters by efficiently fusing LiDAR and camera streams.
  • High-Speed Detection: COMO by Chang Liu achieves speeds exceeding 200 FPS for RGB-Infrared detection, a feat difficult to reach with heavy attention-based "necks."
  • Medical Imaging: In survival analysis, ME-Mamba by Chengsheng Zhang reduced GPU memory usage by 62%, allowing the model to process massive gigapixel pathology slides alongside genomic data.

Beyond Simple Fusion: Temporal Continuity

A recurring theme in recent work, such as STTrack by Xiantao Hu, is that SSMs treat temporal context as a continuous stream rather than a set of discrete frames. This is particularly powerful for sensors with different temporal resolutions, such as standard cameras (30 FPS) and event cameras (microsecond resolution). By using the SSM's hidden state as a persistent memory, models like Mamba-FETrack V2 can "remember" a target's appearance even during total occlusion or rapid motion.

By moving from the "all-to-all" comparison of attention to the "state-to-state" evolution of Mamba, this research direction is enabling a new generation of multimodal systems that are faster, lighter, and more capable of handling the high-dimensional data of the real world.

Papers in this version

Showing 60 of 88 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_complexity_multimodal_fusion_with_4f4cc2be,
  title = {Linear Complexity Multimodal Fusion with State Space Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-complexity-multimodal-fusion-with-state-space-models/rdr_4f4cc2be66ab4102b5bdbf958a9d2fdb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_complexity_multimodal_fusion_with_4f4cc2be_v1,
  title = {Linear Complexity Multimodal Fusion with State Space Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-complexity-multimodal-fusion-with-state-space-models/rdr_4f4cc2be66ab4102b5bdbf958a9d2fdb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}