Rethinking Mamba in Speech Processing by Self-Supervised Models

Xiangyu Zhang, Jianbo Ma, Mostafa Shahin, Beena Ahmed, Julien EppsICASSP 2025 · DOI

TL;DR

This paper investigates why the Mamba architecture, while powerful, shows inconsistent performance across different speech processing tasks. They hypothesize Mamba excels at "reconstruction" tasks but needs additional components for "classification" tasks like speech recognition. By analyzing mutual information, they provide evidence supporting this, showing Mamba's internal information flow changes significantly depending on whether a reconstruction or classification objective is present.

What Problem Does This Paper Address?

The Mamba architecture has emerged as a strong contender in various domains, including speech processing. However, its performance isn't uniform across all speech tasks. For tasks like speech enhancement and spectrum reconstruction, Mamba performs well independently. In contrast, for speech recognition, it often requires supplementary modules (like decoders) to match or surpass established attention-based models. This inconsistency raises a question: what are the underlying reasons for Mamba's varying success, and what are its inherent strengths and weaknesses in the speech domain? The authors aim to bridge this gap by exploring Mamba's behavior through an information-theoretic lens.

Methods

The core of this paper's methodology lies in mutual information (MI) analysis using the Mutual Information Neural Estimation (MINE) framework. They analyze the MI between local input features ($X$) and intermediate layer features ($T_i$), denoted as $I(X; T_i)$. The hypothesis is that for "reconstruction" tasks, $I(X; T_i)$ should exhibit a pattern of decreasing and then increasing as layer depth increases, similar to an autoencoder. For "classification" tasks where Mamba struggles, $I(X; T_i)$ is expected to show a more consistent decrease.

To validate this, they:

  1. Analyze existing models: They examine the $I(X; T_i)$ patterns in the ConBiMamba model for Automatic Speech Recognition (ASR) and the Ssamba model for spectrum reconstruction.
  2. Train a Mamba-based HuBERT model: They replace the Transformer layers in the HuBERT architecture with Mamba layers. This is a key step as HuBERT is a self-supervised model trained on tasks that are more akin to classification. They then analyze the $I(X; T_i)$ patterns in this Mamba-HuBERT model, both independently and when augmented with a downstream Conformer model.

Datasets

  • LibriSpeech100: Used for training and evaluating speech recognition models (ConBiMamba and Mamba-HuBERT).
  • Not explicitly named but implied: Datasets for speech enhancement and spectrum reconstruction tasks where existing Mamba models (like Ssamba) are evaluated.

Key Insights and Ideas

  • Mamba's "Reconstruction" Affinity: The paper strongly suggests that Mamba's architecture is inherently well-suited for tasks that involve reconstructing input data, such as speech enhancement or spectrum reconstruction. This is evidenced by the characteristic "information bottleneck" pattern (decrease then increase in $I(X; T_i)$) observed in these tasks.
  • Classification Requires Augmentation: For tasks like speech recognition, which are more classification-oriented, Mamba alone is insufficient to achieve state-of-the-art performance. The analysis shows that when Mamba is used for ASR without a decoder, its $I(X; T_i)$ trend doesn't align with reconstruction, and performance suffers.
  • The Role of the Decoder: The addition of a decoder (a common component in ASR systems) fundamentally changes Mamba's information flow. When a decoder is present, the $I(X; T_i)$ pattern in Mamba-based ASR models shifts to exhibit the "reconstruction" trend, leading to improved performance. This implies the decoder helps Mamba "reconstruct" the necessary information for classification.
  • HuBERT as a "Classification" Proxy: Using Mamba within the HuBERT framework provides a compelling case. HuBERT's training objective (predicting pseudo-labels) is more classification-like. The Mamba-HuBERT model, when used independently, shows a decreasing $I(X; T_i)$ trend and underperforms standard HuBERT. However, when a Conformer (a downstream model) is added, the $I(X; T_i)$ trend becomes more "reconstruction-like," and performance becomes comparable to HuBERT with a Conformer.

Key Results

  • ConBiMamba (ASR): Without a decoder, $I(X; T_i)$ decreases, leading to suboptimal performance. With a decoder, $I(X; T_i)$ decreases then increases, and performance improves significantly (e.g., WER drops from 8.4 to 5.9 on LibriSpeech100 test-clean).
  • Ssamba (Spectrum Reconstruction): The $I(X; T_i)$ pattern shows a decrease followed by an increase, consistent with reconstruction tasks.
  • Mamba-HuBERT (Pre-training):
    • Independently: $I(X; T_i)$ continuously decreases, showing underperformance compared to standard HuBERT.
  • With Conformer: $I(X; T_i)$ exhibits the decrease-then-increase "reconstruction" pattern, achieving performance comparable to HuBERT with a Conformer (e.g., Mamba-HuBERT trained with Layer 9 features + Conformer achieves 9.2 WER on test-clean, vs. HuBERT + Conformer at 9.3 WER).

Limitations and Future Work

  • Focus on Mutual Information: While MI analysis provides strong evidence, it's a theoretical lens. Direct causal links between MI trends and specific architectural mechanisms could be further explored.
  • Task Specificity: The findings are specific to the speech tasks examined. Further investigation into other speech modalities or cross-modal applications could reveal more about Mamba's generalizability.
  • Future Work: The authors implicitly suggest exploring how to best integrate Mamba into classification pipelines by designing specific auxiliary modules or decoders that leverage Mamba's reconstruction strengths. They also highlight the potential for Mamba in self-supervised learning for speech, especially when combined with appropriate downstream tasks.

Summary updated

BibTeX
@inproceedings{zhang2025rethinking,
  author = {Xiangyu Zhang and Jianbo Ma and Mostafa Shahin and Beena Ahmed and Julien Epps},
  title = {{Rethinking Mamba in Speech Processing by Self-Supervised Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889111},
}
Figure