Bidirectional State Space Models

61 concepts55 papers2 proposals

State Space Models (SSMs), particularly selective architectures like Mamba, have emerged as a powerful alternative to Transformers. By mapping input sequences through a continuous state space, these models achieve linear $O(N)$ computational complexity relative to sequence length, bypassing the quadratic bottleneck of self-attention.

However, standard SSMs inherit a fundamental limitation from their recurrent ancestors: causality. They process information strictly from left to right. While this one-way conveyor belt is ideal for autoregressive language generation, it introduces a severe architectural mismatch when applied to non-causal domains like genomics, audio processing, and computer vision, where context from both the "past" and the "future" is vital. As noted by Tri Dao et al. in Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, adapting linear-time state space duality frameworks to non-causal architectures remains an active frontier.

To bridge this gap, researchers have developed Bidirectional State Space Models. By restructuring how the state space is scanned, these architectures capture global, multi-directional context without sacrificing linear scaling.

The Core Mechanism: Dual-Path Scanning

The core intuition behind a bidirectional SSM is simple: if a single state space scan can only look backward, running a second scan in reverse allows the model to look forward.

Forward Scan:  [Token 1] ---> [Token 2] ---> [Token 3] ---> Hidden State (Forward)
Backward Scan: [Token 1] <--- [Token 2] <--- [Token 3] <--- Hidden State (Backward)
                                                                   |
                                                            Combined Context

To implement this without losing the efficiency of the underlying hardware-aware parallel scans, models typically employ one of two structural configurations:

Parallel Scanning

The input sequence is fed simultaneously into two independent SSM paths. One path processes the sequence in its original chronological order, while the other processes a reversed copy of the sequence. As described by Jiangyan Yi et al. in RawBMamba: End-to-End Bidirectional State Space Model for Audio Deepfake Detection, this dual-direction parallel processing ensures that the resulting representations incorporate global context from both ends of a temporal signal.

Series Scanning

Rather than running the scans side-by-side, some architectures chain them together. Zheyu Zhang et al. demonstrate in Incomplete Multi-modal Brain Tumor Segmentation via Learnable Sorting State Space Model that connecting forward and backward state space scans in a series configuration forces a deeper integration of information across the entire data sequence, which is particularly effective for capturing intricate local textures alongside long-range relationships.

Integrating the Paths

Once both directions are scanned, their hidden states must be merged. A naive approach might share parameters between the paths, but direction-specific dynamics usually differ. Xiangyu Zhang et al. show in Mamba in Speech: Towards an Alternative to Self-Attention that utilizing separate linear projections for the forward and backward paths allows the model to learn direction-specific features more effectively.

After projection, these representations are typically combined and refined. Rong Chao et al. in An Investigation of Incorporating Mamba for Speech Enhancement demonstrate that concatenating the forward and backward representations and passing them through a convolutional layer provides a comprehensive, non-causal view of the sequence while maintaining a lower computational footprint than self-attention.

Domain Symmetries: Why Bidirectionality Matters

The necessity of bidirectional scanning becomes obvious when looking at fields where data lacks a natural arrow of time.

Genomics and RNA Language Models

In genetics, DNA and RNA strands are inherently non-causal. Regulatory elements can influence gene expression from hundreds of thousands of base pairs away, whether they are located upstream or downstream. In Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling, the authors emphasize that standard uni-directional sequence models fail to naturally account for this non-causal biological structure.

By embedding bidirectionality directly into the state space architecture, genomic foundation models like PlantBiMoE (developed by Kepeng Lin et al.) and HydraRNA (developed by Guipeng Li et al.) can capture structural dependencies from both strands of a sequence simultaneously, scaling linearly over massive context windows.

Audio and Speech Processing

Audio signals are highly continuous and contextual; a localized acoustic glitch or deepfake anomaly can often only be identified by evaluating the surrounding rhythmic and phonetic context. Arda Senocak et al. demonstrate in Audio Mamba: Bidirectional State Space Model for Audio Representation Learning that bidirectional scanning allows state-space models to successfully process two-dimensional non-causal data like audio spectrograms.

For generative audio tasks, such as diffusion-based synthesis, Karan Goel et al. show in It's Raw! Audio Generation with State-Space Models that bidirectional variants provide the global context necessary to drastically improve the quality of the iterative denoising process.

Alternative Non-Causal Formulations

While running parallel forward and backward scans is the most prominent strategy, researchers have explored other creative ways to introduce non-causality into sub-quadratic sequence models:

By breaking the constraint of strict causality while preserving the mathematical elegance of the state space abstraction, bidirectional SSMs provide a scalable, linear-time foundation for processing the inherently non-causal structures of the physical and biological world.

Key Foundational Work

A Neural Multi-Sequence Alignment TeCHnique (NeuMATCH) (2018) — contributed 1 concept elements to this direction.
Implicitly-Defined Neural Networks for Sequence Labeling (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_state_space_modeling_7b8f53a3,
  title = {bidirectional state space modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-state-space-modeling/rdr_7b8f53a3474f4012be8ed5162dc86809},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_state_space_modeling_7b8f53a3_v1,
  title = {bidirectional state space modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-state-space-modeling/rdr_7b8f53a3474f4012be8ed5162dc86809/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake