An Exploration of Mamba for Speech Self-Supervised Models

Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun-Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee

An Exploration of Mamba for Speech Self-Supervised Models

Modern speech self-supervised learning (SSL) models, such as wav2vec 2.0 and HuBERT, have traditionally relied on Transformer architectures. While effective, Transformers suffer from quadratic computational complexity relative to sequence length. This poses a significant bottleneck for audio processing, where sequences can be exceptionally long. This paper investigates the potential of Mamba, a Selective State Space Model with linear scaling, to replace Transformers within the HuBERT framework.

Methodology

The researchers constructed Mamba-based HuBERT models by substituting standard Transformer blocks with Mamba blocks. The study evaluated these models in two primary configurations:

  1. Causal (Unidirectional): Essential for real-time or streaming automatic speech recognition (ASR).
  2. Bidirectional: Utilizing specialized variants like External Bidirectional Mamba (ExtBiMamba) for offline processing.

Computational Efficiency and Scaling

The most significant advantage of the Mamba architecture is its computational efficiency. Unlike Transformers, which exhibit quadratic growth in compute requirements and memory usage, Mamba scales linearly.

Mamba demonstrates constant compute cost (MACs/sec) as sequence length increases, whereas Transformers encounter Out-of-Memory (OOM) errors beyond 80 seconds.

As illustrated in the figure above, the compute cost (MACs/second) for Mamba remains flat as audio length increases from 5 seconds to over 5 minutes. In contrast, the Transformer's compute requirements skyrocket, leading to Out-of-Memory (OOM) failures at approximately 80 seconds. This makes Mamba uniquely suited for processing long-context audio, such as full meetings or podcasts, without the need for aggressive segmentation.

Representation Quality

Beyond efficiency, the study analyzed the quality of the learned speech representations. Using "phone purity" metrics, the authors found that Mamba's hidden layers map more cleanly to human phonemes than Transformer layers do.

Phone purity across layers for various architectures. Mamba-based models, particularly ExtBiMamba, achieve high phonetic alignment in the middle-to-late layers.

Furthermore, Canonical Correlation Analysis (CCA) revealed that Mamba captures speaker-related features more distinctly in its early layers. This suggests that Mamba is not just a faster alternative, but one that may extract superior phonetic and speaker-related information for downstream tasks.

Key Results

  • Long-Context ASR: Fine-tuning ExtBiMamba on unsegmented documents reduced the Word Error Rate (WER) from 13.37% to 11.08%. The Transformer baseline was unable to run this task due to memory constraints.
  • Streaming ASR: A 78M parameter causal Mamba achieved a 15.77% WER, outperforming a larger 94M parameter causal Transformer (16.66% WER).
  • SUPERB Benchmark: In causal probing, the Mamba+MLP model achieved a score of 823.15, significantly higher than the Causal Transformer's 805.44.

Conclusion

Mamba represents a promising shift for speech SSL, offering linear scaling that unlocks long-context processing while maintaining or exceeding the representational power of Transformers. While bidirectional scaling at larger model sizes remains an area for architectural refinement, Mamba's inherent causal efficiency makes it a natural fit for the next generation of streaming and large-scale speech foundation models.

Summary updated

BibTeX
@misc{lin2025exploration,
  author = {Tzu-Quan Lin and Heng-Cheng Kuo and Tzu-Chieh Wei and Hsi-Chun Cheng and Chun-Wei Chen and Hsien-Fu Hsiao and Yu Tsao and Hung-yi Lee},
  title = {{An Exploration of Mamba for Speech Self-Supervised Models}},
  year = {2025},
  eprint = {2506.12606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.12606},
}
Figure