Multimodal models must find a way to let different data streams—such as text, images, audio, or asynchronous sensor events—talk to each other. Traditionally, this conversation happens via cross-attention or joint self-attention in Transformers. To fuse an image with a text prompt, the model computes an attention matrix mapping every visual token to every textual token.
While highly effective, this approach suffers from a fundamental flaw: quadratic complexity. If the sequence length of the input modalities increases, the computational and memory requirements grow quadratically ($O(N^2)$). This "memory wall" makes it prohibitively expensive to process high-frequency or long-form multimodal data, such as long video sequences, high-resolution medical time-series, or dense 3D point clouds.
The Linear Alternative: Selective State Space Models
To bypass this bottleneck, a rapidly growing body of research replaces attention mechanisms with selective State Space Models (SSMs), most notably the Mamba architecture. Unlike Transformers, which look back at all historical tokens at every step, SSMs compress past information into a fixed-size hidden state $\mathbf{h}_t$. The system updates this state sequentially via linear operators:
$$\mathbf{h}_t = \mathbf{A}_t \mathbf{h}_{t-1} + \mathbf{B}_t \mathbf{x}_t$$ $$\mathbf{y}_t = \mathbf{C}_t \mathbf{h}_t$$
Because the computation at each step only depends on the current input $\mathbf{x}_t$ and the previous hidden state $\mathbf{h}_{t-1}$, the computational complexity scales linearly ($O(N)$) with the total sequence length. As demonstrated by Zhao et al. and Wu, this architectural shift allows Multi-Modal Large Language Models (MLLMs) to achieve massive speedups in token generation and inference throughput without sacrificing reasoning quality.
Mechanics of State-Space Fusion
How do you force two entirely different modalities to interact inside a sequential state space without the explicit all-to-all matching of cross-attention? Researchers have converged on three primary architectural paradigms to achieve this:
1. Token Interleaving (Sequence Weaving)
The most direct way to fuse modalities is to project them into a shared embedding space and interleave their tokens into a single sequence. For example, in referring image segmentation, Mao et al. repeatedly interleave textual and visual tokens. As the selective scan moves through this unified sequence, the hidden state $\mathbf{h}_t$ acts as a continuous memory bridge. When the scan transitions from a visual patch to a text phrase, the hidden state carries the contextual visual information forward, updating its representation dynamically. Similarly, Ke et al. show that interleaving spatial queries and image features allows SSMs to learn complex Bird's-Eye-View (BEV) representations for autonomous driving.
2. Cross-Branch Hidden State Exchange
Instead of mixing tokens into a single stream, some architectures maintain distinct processing branches for each modality but allow them to communicate during the scanning process. In event-based video super-resolution, the authors of Xiao et al. process standard RGB frames and asynchronous event streams in parallel branches. Rather than using heavy cross-attention layers, the branches exchange their hidden states during the sequential scan. This allows high-frequency motion cues from the event camera to selectively update the structural state of the RGB frame branch, maintaining linear complexity while integrating asynchronous sensor data. Sun et al. apply a similar bidirectional state-exchange mechanism to track objects across modalities efficiently.
3. Shared State Transition Matrices
A deeper level of fusion involves coupling the mathematical parameters that govern how the state space evolves over time. Ye et al. introduced a progressive fusion strategy for audio-visual depression detection where the state transition matrices ($\mathbf{A}$) are shared across different input streams. This means that while each modality retains its own unique projections, the underlying temporal dynamics and the rate at which information is accumulated or forgotten are co-determined by both modalities.
Unlocking All-Layer Dense Interaction
In traditional Transformer architectures, the high computational cost of cross-attention forces designers to be stingy: they typically fuse modalities only at the very beginning (early fusion) or at the very end (late fusion) of the network.
Linear-complexity SSMs completely change this design constraint. Because the memory overhead is so low, models can afford to perform cross-modal fusion across every single layer of a deep network. For instance, the authors of RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba leverage this property to enable dense, continuous interaction between visible and thermal features throughout the entire backbone. This allows the network to simultaneously integrate low-level textures and high-level semantics without hitting a memory wall.
Broad Applicability and Real-Time Control
The efficiency gains of state-space fusion have unlocked real-time performance in domains where Transformers or diffusion models are too slow. In robotics, Nguyen et al. and Wang et al. demonstrate that treating combined vision, point cloud, and language data as a sequence within an SSM framework enables real-time control and language-driven grasp detection, executing hundreds of times faster than diffusion-based alternatives.
From high-throughput material design (Gu et al.) to multi-camera target tracking (Fan et al.) and modular biomedical model merging (Hemker), replacing quadratic attention with selective state space updates provides a scalable, unified path forward for multimodal AI.
Key Foundational Work
Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation (2025) — contributed 2 concept elements to this direction.
Exploring Historical Information for RGBE Visual Tracking with Mamba (2025) — contributed 2 concept elements to this direction.
VL-Mamba: Exploring State Space Models for Multimodal Learning (2024) — contributed 2 concept elements to this direction.