TL;DR
MLMA (Multilingual Language Modeling with Mamba for ASR) is the first multilingual automatic speech recognition system built on the efficient Mamba state-space architecture. By processing nearly 12,000 hours of speech across six European languages, MLMA demonstrates that Mamba-based models can match or outperform traditional Transformer-based architectures while offering a more scalable, memory-efficient alternative.
What Problem Does This Paper Address?
While automatic speech recognition (ASR) has reached near-human performance for high-resource languages like English, scaling these systems to be truly multilingual remains a challenge. Training a single model on multiple languages allows it to exploit shared phonetic and acoustic representations, which helps boost performance on lower-resource languages.
However, the current dominant architecture—the Transformer—suffers from quadratic computational and memory complexity relative to sequence length. This inefficiency becomes a major bottleneck in multilingual scenarios where models must process highly variable speech rates, tonal variations, and code-switching over long utterances.
Prior attempts to use the highly efficient Mamba architecture (a state-space model with linear-time complexity) for ASR were strictly limited to monolingual or bilingual setups on small datasets. This paper bridges that gap by evaluating Mamba's capabilities in a large-scale, multilingual framework.
Methods
The authors introduce the ConMamba (Convolutional Mamba) Encoder, which replaces the standard multi-head self-attention mechanism found in Transformers with bidirectional Mamba (BiMamba) layers, while retaining convolutional layers to handle local feature extraction.
The overall MLMA architecture operates within a Connectionist Temporal Classification (CTC) framework:
- Feature Extraction: Input audio is converted into 80-dimensional log Mel filter banks and downsampled using a two-block CNN.
- Context Modeling: An 18-layer encoder structure models contextual representations, augmented with a BiMamba module to capture long-range dependencies efficiently.
- Decoding: A linear projection and LogSoftmax layer map the encoder outputs to the target text vocabulary, trained via CTC loss.
Datasets
The model was trained on a massive collection of over 11,000 hours of labeled speech data spanning six European languages: English (en), Italian (it), French (fr), Spanish (es), German (de), and Dutch (nl). The data was sourced from four major corpora:
- LibriSpeech (LS) (clean subsets)
- CommonVoice v20.0 (CV)
- VoxPopuli-ASR (VP)
- MultiLingual LibriSpeech (MLS)
For out-of-domain evaluation, the authors used the FLEURS (FL) benchmark, which was completely unseen during the training phase.
Key Insights and Ideas
- Mamba Scales to Multilingual Workloads: The paper proves that Mamba's sequential inductive bias is fully capable of learning shared, robust cross-lingual linguistic representations, dispelling doubts that state-space models might struggle outside of monolingual domains.
- Effective Hybrid Design: Combining convolution (for local acoustic patterns) with Mamba (for global context) provides a strong architectural synergy for speech processing, yielding lower Word Error Rates (WER) than standard Conformers.
- Data Efficiency: Despite being significantly smaller and using a simplified training setup compared to massive foundation models (like Whisper or OWSM), MLMA achieves highly competitive performance.
Key Results
- Monolingual Baseline: When trained from scratch on LibriSpeech-1000, ConMamba achieved a WER of 4.05% on test-clean and 10.50% on test-other, outperforming the standard Conformer baseline (4.27% and 11.29%, respectively).
- Multilingual Performance: On the in-domain MLS dataset, MLMA consistently outperformed the OWSM-CTC foundation model across multiple languages, such as achieving a 9.1% WER in French compared to OWSM-CTC's 12.9%.
- Out-of-Domain Robustness: Evaluation on the unseen FLEURS benchmark demonstrated stable, competitive performance, highlighting the model's ability to generalize well to new acoustic environments.
- Model Scaling: Increasing the ConMamba parameter count from 31.6M to 42M yielded significant WER reductions, particularly on phonetically rich languages like English (dropping from 23.00% to 21.04% on CommonVoice).
Limitations and Future Work
While MLMA shows remarkable scalability, performance still varies noticeably by language. For instance, during bilingual scaling experiments, increasing the training data size yielded massive out-of-domain gains for English on the FLEURS benchmark, but this trend was not mirrored for Italian. The authors attribute this to the inherent data imbalance between the languages in the training set.
Future work will likely focus on addressing language imbalance during training and scaling the architecture to match the parameter sizes of massive billion-parameter foundation models.
Summary updated