MLMA: Towards Multilingual ASR With Mamba-based Architectures

Mohamed Nabih Ali, Daniele Falavigna, Alessio BruttiarXiv 2025 · arXiv

TL;DR

MLMA (Multilingual Language Modeling with Mamba for ASR) is the first multilingual automatic speech recognition system built on the efficient Mamba state-space architecture. By processing nearly 12,000 hours of speech across six European languages, MLMA demonstrates that Mamba-based models can match or outperform traditional Transformer-based architectures while offering a more scalable, memory-efficient alternative.

What Problem Does This Paper Address?

While automatic speech recognition (ASR) has reached near-human performance for high-resource languages like English, scaling these systems to be truly multilingual remains a challenge. Training a single model on multiple languages allows it to exploit shared phonetic and acoustic representations, which helps boost performance on lower-resource languages.

However, the current dominant architecture—the Transformer—suffers from quadratic computational and memory complexity relative to sequence length. This inefficiency becomes a major bottleneck in multilingual scenarios where models must process highly variable speech rates, tonal variations, and code-switching over long utterances.

Prior attempts to use the highly efficient Mamba architecture (a state-space model with linear-time complexity) for ASR were strictly limited to monolingual or bilingual setups on small datasets. This paper bridges that gap by evaluating Mamba's capabilities in a large-scale, multilingual framework.

Table 1. The table summaries recent works exploring Mamba for ASR.

Methods

The authors introduce the ConMamba (Convolutional Mamba) Encoder, which replaces the standard multi-head self-attention mechanism found in Transformers with bidirectional Mamba (BiMamba) layers, while retaining convolutional layers to handle local feature extraction.

The overall MLMA architecture operates within a Connectionist Temporal Classification (CTC) framework:

  1. Feature Extraction: Input audio is converted into 80-dimensional log Mel filter banks and downsampled using a two-block CNN.
  2. Context Modeling: An 18-layer encoder structure models contextual representations, augmented with a BiMamba module to capture long-range dependencies efficiently.
  3. Decoding: A linear projection and LogSoftmax layer map the encoder outputs to the target text vocabulary, trained via CTC loss.
Fig. 1. The architecture of MLMA using ConMamba encoder and a CTC decoder as in [20]

Datasets

The model was trained on a massive collection of over 11,000 hours of labeled speech data spanning six European languages: English (en), Italian (it), French (fr), Spanish (es), German (de), and Dutch (nl). The data was sourced from four major corpora:

  • LibriSpeech (LS) (clean subsets)
  • CommonVoice v20.0 (CV)
  • VoxPopuli-ASR (VP)
  • MultiLingual LibriSpeech (MLS)

For out-of-domain evaluation, the authors used the FLEURS (FL) benchmark, which was completely unseen during the training phase.

Table 2. List of training data used in our experiments.

Key Insights and Ideas

  • Mamba Scales to Multilingual Workloads: The paper proves that Mamba's sequential inductive bias is fully capable of learning shared, robust cross-lingual linguistic representations, dispelling doubts that state-space models might struggle outside of monolingual domains.
  • Effective Hybrid Design: Combining convolution (for local acoustic patterns) with Mamba (for global context) provides a strong architectural synergy for speech processing, yielding lower Word Error Rates (WER) than standard Conformers.
  • Data Efficiency: Despite being significantly smaller and using a simplified training setup compared to massive foundation models (like Whisper or OWSM), MLMA achieves highly competitive performance.

Key Results

  • Monolingual Baseline: When trained from scratch on LibriSpeech-1000, ConMamba achieved a WER of 4.05% on test-clean and 10.50% on test-other, outperforming the standard Conformer baseline (4.27% and 11.29%, respectively).
Table 3. WER of ConMamba and Conformer on LibriSpeech dataset. Results are similar to what reported in [20]
  • Multilingual Performance: On the in-domain MLS dataset, MLMA consistently outperformed the OWSM-CTC foundation model across multiple languages, such as achieving a 9.1% WER in French compared to OWSM-CTC's 12.9%.
  • Out-of-Domain Robustness: Evaluation on the unseen FLEURS benchmark demonstrated stable, competitive performance, highlighting the model's ability to generalize well to new acoustic environments.
Table 5. WER(↓%) of MLMA across multilingual data. The numbers of OSWM-CTC are from [39]. FL*: FLEURS is not used in training. ”-”: results not reported in the reference paper.
  • Model Scaling: Increasing the ConMamba parameter count from 31.6M to 42M yielded significant WER reductions, particularly on phonetically rich languages like English (dropping from 23.00% to 21.04% on CommonVoice).
Table 6. WER(%) (↓) on CV scaling the size of MLMA

Limitations and Future Work

While MLMA shows remarkable scalability, performance still varies noticeably by language. For instance, during bilingual scaling experiments, increasing the training data size yielded massive out-of-domain gains for English on the FLEURS benchmark, but this trend was not mirrored for Italian. The authors attribute this to the inherent data imbalance between the languages in the training set.

Future work will likely focus on addressing language imbalance during training and scaling the architecture to match the parameter sizes of massive billion-parameter foundation models.

Summary updated

Abstract

Multilingual automatic speech recognition (ASR) remains a challenging task, especially when balancing performance across high- and low-resource languages. Recent advances in sequence modeling suggest that architectures beyond Transformers may offer better scalability and efficiency. In this work, we introduce MLMA (Multilingual Language Modeling with Mamba for ASR), a new approach that leverages the Mamba architecture -- an efficient state-space model optimized for long-context sequence processing -- for multilingual ASR. Using Mamba, MLMA implicitly incorporates language-aware conditioning and shared representations to support robust recognition across diverse languages. Experiments on standard multilingual benchmarks show that MLMA achieves competitive performance compared to Transformer-based architectures. These results highlight Mamba's potential as a strong backbone for scalable, efficient, and accurate multilingual speech recognition.
BibTeX
@misc{ali2025mlma,
  author = {Mohamed Nabih Ali and Daniele Falavigna and Alessio Brutti},
  title = {{MLMA: Towards Multilingual ASR With Mamba-based Architectures}},
  year = {2025},
  eprint = {2510.18684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.18684},
}
Figure