Hardware Optimization for Linear State Space Models

51 concepts49 papers1 proposal

State Space Models (SSMs) promise a computational revolution in sequence modeling: replacing the $O(N^2)$ quadratic bottleneck of the Transformer's attention mechanism with $O(N)$ linear complexity. In theory, this allows models to process massive contexts, high-resolution images, or long-horizon robotic trajectories with minimal computational scaling.

However, a persistent paradox emerges when deploying these architectures: models with significantly lower theoretical floating-point operations (FLOPs) often run slower than their attention-based counterparts on actual hardware. As identified by Lu et al. in their work on multimodal tracking, this discrepancy stems from a profound hardware-software optimization gap. While the mathematics of SSMs are elegant, modern computing infrastructure is fundamentally biased toward a different class of operations.

The Hardware Bias: Compute-Bound vs. Memory-Bound Operations

To understand why this gap exists, we must look at how modern hardware accelerators like GPUs and Tensor Processing Units (TPUs) are engineered. These chips are optimized for dense matrix multiplications (GEMM)—the exact operation that powers convolutional neural networks (CNNs) and Transformers. Over the past decade, these architectures have benefited from highly mature, low-level software libraries (such as FlashAttention) that maximize hardware utilization by keeping data inside fast on-chip SRAM memory.

SSMs operate differently. Instead of massive parallel matrix multiplications, they rely on recurrent, sequential updates to a hidden state. As noted by Avenstrup et al., current implementations of SSMs are frequently limited by memory bandwidth rather than raw computational power (Avenstrup et al.). The hardware spends more time moving hidden states between slow High-Bandwidth Memory (HBM) and fast on-chip SRAM than it does performing actual arithmetic.

According to Huang et al., deep learning frameworks remain heavily biased toward dense matrix operations, leaving the inherently serial scanning processes of SSMs at a practical latency disadvantage (Huang et al.). As Y. Isabel Liu notes, emerging neural architectures face a significant performance disadvantage because they lack the years of specialized kernel development that established models enjoy (Y. Isabel Liu).

The Overhead of Multi-Dimensional Scanning

The optimization gap widens dramatically when moving from 1D text sequences to multi-dimensional data, such as 2D images or 3D point clouds. Because standard SSMs are inherently causal and one-dimensional, processing spatial data requires flattening the inputs and scanning them in multiple directions to capture global context without a directional bias.

Standard 1D Scan:   [A] ---> [B] ---> [C] ---> [D]  (Efficient, sequential)

Multi-Directional:  [A] ---> [B]      [A] <--- [B]  (Requires data duplication,
                    |        |        |        |   reshaping, and multiple
                    v        v        ^        ^   memory round-trips)
                    [C] ---> [D]      [C] <--- [D]

This multi-directional scanning introduces severe computational friction. Shi et al. demonstrate that strategies like scanning across six distinct directions incur massive data reshaping and permutation overhead (Shi et al.). Every time a sequence is re-ordered or transposed to accommodate a new scanning path, the hardware must execute memory copies that stall the execution pipeline.

The authors of another study emphasize that without specialized low-level kernels designed to handle these non-causal spatial scans, the throughput of the model plummets, erasing the theoretical benefits of linear complexity (Chng et al.).

The Portability Barrier and Edge Deployment

To achieve competitive wall-clock speeds, early SSM implementations relied on highly specialized, hardware-specific CUDA kernels. However, this creates a major engineering bottleneck: portability.

As argued by Deng et al., relying on platform-specific CUDA kernels prevents these models from being easily deployed on diverse architectures, such as mobile Neural Processing Units (NPUs) or edge devices (Deng et al.). Wu et al. echo this concern, noting that the lack of mature hardware acceleration on edge hardware prevents theoretical linear efficiency from translating into real-world performance for semantic communications (Wu et al.).

When deploying on mobile CPUs, He et al. observe a stark performance gap compared to CNNs, driven entirely by the novelty of the architecture and the absence of a decade's worth of low-level compiler optimizations (He et al.).

Bridging the Gap: Algorithmic and Architectural Solutions

To bridge this gap, researchers are attacking the problem from both the algorithmic and kernel-design levels, reshaping how models interact with hardware memory.

Frequency-Domain Parameterization

One elegant approach bypasses specialized spatial kernels entirely by changing the mathematical domain of the operation. Parnichkun et al. introduce a "state-free" parameterization of SSMs using Rational Transfer Functions (RTFs) in the frequency domain (Parnichkun et al.). By decoupling the hidden state size from the computational cost of parallel inference, they eliminate the memory bloat and platform-specific kernel dependencies that plague standard time-domain implementations.

Streamlined Scanning Patterns

Other approaches modify the scanning mechanism itself to be more hardware-friendly. For instance, Li et al. propose an additive linear recurrence mechanism in LightNet that processes multi-dimensional data in a single sequential scan, minimizing the need for multiple memory-disruptive passes (Li et al.).

Similarly, for high-dimensional 3D perception, Li et al. utilize a height-prioritized 2D Hilbert expansion to linearize 3D voxels efficiently, reducing memory overhead without stalling standard hardware accelerators (Li et al.).

Alternative Linear Architectures

Rather than fixing the scanning kernels of SSMs, some researchers explore entirely different mathematical formulations to achieve linear complexity without recurrence. Peng et al. propose the Star with Bilinear Mapping (SBM) architecture, which replaces both attention and recurrent scans with element-wise multiplications and bilinear mappings (Peng et al.). This allows the model to bypass the hardware-specific CUDA kernel dependencies of SSMs entirely, achieving high practical throughput on standard deep learning frameworks.

Key Foundational Work

LocalMamba: Visual State Space Model with Windowed Selective Scan (2024) — contributed 2 concept elements to this direction.
MambaJSCC: Deep Joint Source-Channel Coding with Visual State Space Model (2024) — contributed 1 concept elements to this direction.
DVMSR: Distillated Vision Mamba for Efficient Super-Resolution (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hardware_software_optimization_gap_e938f85f,
  title = {hardware-software optimization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-software-optimization-gap/rdr_e938f85f5ea54f9c8664db7a8bedaa64},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hardware_software_optimization_gap_e938f85f_v1,
  title = {hardware-software optimization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-software-optimization-gap/rdr_e938f85f5ea54f9c8664db7a8bedaa64/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake