An Exploration of Mamba for Speech Self-Supervised Models
Modern speech self-supervised learning (SSL) models, such as wav2vec 2.0 and HuBERT, have traditionally relied on Transformer architectures. While effective, Transformers suffer from quadratic computational complexity relative to sequence length. This poses a significant bottleneck for audio processing, where sequences can be exceptionally long. This paper investigates the potential of Mamba, a Selective State Space Model with linear scaling, to replace Transformers within the HuBERT framework.
Methodology
The researchers constructed Mamba-based HuBERT models by substituting standard Transformer blocks with Mamba blocks. The study evaluated these models in two primary configurations:
- Causal (Unidirectional): Essential for real-time or streaming automatic speech recognition (ASR).
- Bidirectional: Utilizing specialized variants like External Bidirectional Mamba (ExtBiMamba) for offline processing.
Computational Efficiency and Scaling
The most significant advantage of the Mamba architecture is its computational efficiency. Unlike Transformers, which exhibit quadratic growth in compute requirements and memory usage, Mamba scales linearly.
As illustrated in the figure above, the compute cost (MACs/second) for Mamba remains flat as audio length increases from 5 seconds to over 5 minutes. In contrast, the Transformer's compute requirements skyrocket, leading to Out-of-Memory (OOM) failures at approximately 80 seconds. This makes Mamba uniquely suited for processing long-context audio, such as full meetings or podcasts, without the need for aggressive segmentation.
Representation Quality
Beyond efficiency, the study analyzed the quality of the learned speech representations. Using "phone purity" metrics, the authors found that Mamba's hidden layers map more cleanly to human phonemes than Transformer layers do.
Furthermore, Canonical Correlation Analysis (CCA) revealed that Mamba captures speaker-related features more distinctly in its early layers. This suggests that Mamba is not just a faster alternative, but one that may extract superior phonetic and speaker-related information for downstream tasks.
Key Results
- Long-Context ASR: Fine-tuning ExtBiMamba on unsegmented documents reduced the Word Error Rate (WER) from 13.37% to 11.08%. The Transformer baseline was unable to run this task due to memory constraints.
- Streaming ASR: A 78M parameter causal Mamba achieved a 15.77% WER, outperforming a larger 94M parameter causal Transformer (16.66% WER).
- SUPERB Benchmark: In causal probing, the Mamba+MLP model achieved a score of 823.15, significantly higher than the Causal Transformer's 805.44.
Conclusion
Mamba represents a promising shift for speech SSL, offering linear scaling that unlocks long-context processing while maintaining or exceeding the representational power of Transformers. While bidirectional scaling at larger model sizes remains an area for architectural refinement, Mamba's inherent causal efficiency makes it a natural fit for the next generation of streaming and large-scale speech foundation models.
Summary updated