Optimization Brittleness of State Space Models

13 concepts9 papers1 proposal

State Space Models (SSMs) have emerged as a powerful alternative to Transformers, offering linear scaling and efficient long-range dependency modeling. However, their adoption is complicated by a recurring practical challenge: they are significantly more difficult to optimize than the architectures they aim to replace. While Transformers are famously robust to a wide range of training configurations, SSMs often exhibit extreme sensitivity to hyperparameters, particularly the learning rate.

The Learnability Crisis

The performance gap often observed between SSMs and Transformers in tasks like associative recall may not be a result of limited expressive power, but rather a "learnability" crisis. Orvieto and Okpekpe argue that while SSMs like Mamba possess the theoretical capacity to solve complex sequence tasks, they are incredibly sensitive to learning rate selection.

Unlike Transformers, which maintain stable performance across a broad spectrum of optimization settings, SSMs require precise tuning within a narrow window. If the learning rate is slightly outside this optimal range, performance can drop to near-zero. This optimization brittleness, as characterized by Okpekpe and Orvieto, suggests that many previous benchmarks comparing recurrent models to Transformers may have been confounded by suboptimal tuning rather than inherent architectural limitations.

Dimensions of Sensitivity

The sensitivity of SSMs extends beyond the learning rate to include initialization and structural configuration:

  • Initialization and State Size: The performance of structured SSMs is highly dependent on how the state matrices are initialized. Bonassi et al. note that because these models function as "deep Wiener models," their stability and ability to capture nonlinear benchmarks rely heavily on the careful selection of state size and initialization methods. Similarly, Zhang et al. found that specific configuration choices, such as the number of parallel heads per layer, are critical for achieving optimal results in time-series modeling.
  • Architectural Heuristics: In domains like motion generation, Zhang et al. point out that there is no universal heuristic for determining the ideal number of layers or latent dimensions. This lack of a standard "recipe" means models must be meticulously recalibrated for every new dataset.
  • Stability Mechanisms: To match the stability of simpler baselines in reinforcement learning, Jiang et al. found that advanced sequence models often require specific architectural additions, such as layer normalization, to prevent training divergence.

The Challenge of Fair Comparison

This sensitivity makes it difficult to establish a fair baseline for research. Wiemerslage et al. quantified this sensitivity in character-level tasks, finding that while Transformers are indeed more sensitive than LSTMs, the difference is often exaggerated in literature because LSTMs are rarely tuned with the same intensity.

For SSMs, the stakes are higher. Krinner et al. and Wang et al. both emphasize that the increased complexity of the tuning process makes deployment difficult, requiring specialized expertise to ensure that a model's efficiency gains aren't negated by the cost of finding a working configuration.

Preserving Pre-trained Dynamics

The brittleness of SSMs also poses a challenge for fine-tuning. Because the sequence-processing dynamics of these models are so finely balanced, directly updating core parameters can interfere with their foundational capabilities. To address this, Lee et al. proposed "Memba," a parameter-efficient fine-tuning method that avoids touching the fragile pre-trained core. By injecting bio-inspired "membrane" neurons into the gating mechanism, the authors were able to adapt the model to new tasks without compromising the established temporal dynamics of the SSM.

Understanding the mathematical reasons behind this optimization brittleness remains an open question. As the field moves toward more efficient architectures, bridging this "learnability gap" will be essential for making SSMs as reliable and accessible as the Transformers they seek to succeed.

Key Foundational Work

minimax: Efficient Baselines for Autocurricula in JAX (2023) — contributed 2 concept elements to this direction.
PyramidMamba: Rethinking Pyramid Feature Fusion with Selective Space State Model for Semantic Segmentation of Remote Sensing Imagery (2024) — contributed 1 concept elements to this direction.
Effectively Modeling Time Series with Simple Discrete State Spaces (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hyperparameter_sensitivity_learning_rate_bfd0830e,
  title = {hyperparameter sensitivity (learning rate sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hyperparameter-sensitivity-learning-rate-sensitivity/rdr_bfd0830eab4c446a91bd8bbb7934894b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hyperparameter_sensitivity_learning_rate_bfd0830e_v1,
  title = {hyperparameter sensitivity (learning rate sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hyperparameter-sensitivity-learning-rate-sensitivity/rdr_bfd0830eab4c446a91bd8bbb7934894b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake