Configurable Multilingual ASR with Speech Summary Representations

Harrison Zhu, Ivan Fung, Yingke Zhu, Lahiru SamarakoonarXiv 2024 · DOI · arXiv

Introduction

Building a single Automatic Speech Recognition (ASR) model that handles multiple languages is a balancing act. If you simply pool all data together, the model often suffers from "language confusion," where features from one language interfere with another. Existing solutions like LIDConcat or Framewise Mixture of Experts (MoE) often struggle with either ignoring prompts or degrading local acoustic features by forcing them to carry global language information.

The researchers at Fano have introduced csvMASR, a novel multilingual speech recognition architecture that uses a "Summary Vector" to intelligently route audio through language-specific adapters. By offloading global language identification to this dedicated vector—inspired by BERT’s [CLS] token—the model achieves a 9.95% Word Error Rate (WER) on the Multilingual Librispeech dataset, outperforming standard baselines while remaining highly "configurable" to specific language prompts.

Methods

The core innovation is the Configurable MASR with Summary Vector (csvMASR). Its architecture is built on a Conformer backbone with several key additions designed to separate acoustic modeling from language identification.

The csvMASR architecture featuring the Speech Summary Vector ($\theta_{SV}$) and language-specific adapters.
  • Speech Summary Vector ($\theta_{SV}$): Inspired by "registers" in Vision Transformers and the [CLS] token in BERT, the authors append a learnable vector to the input sequence. Crucially, this vector skips all convolutions but participates in the attention mechanisms. This allows it to gather global context (the "what language is this?") without being muddied by local acoustic patterns.
  • Parameter-Efficient Adapters: Instead of duplicating the whole model for every language, the architecture uses small, language-specific "adapters" inserted into the feed-forward layers.
  • Utterance-Level Weighting: The Summary Vector is passed through a classifier to generate weights ($\alpha$). These weights determine how much each language-specific adapter contributes to the final output.
  • Auxiliary Loss: The model is trained with an extra language classification loss, forcing the Summary Vector to become an expert at identifying the spoken language.

Datasets

The model was evaluated using the Multilingual Librispeech (MLS) dataset, focusing on seven non-English languages: Dutch, French, German, Spanish, Italian, Portuguese, and Polish. English was excluded to prevent its massive size (44k hours) from overwhelming the other languages, allowing for a clearer measurement of multilingual interference and adapter effectiveness.

Key Insights

  • Separation of Concerns: By using a Summary Vector, the model effectively separates "what is being said" (frame-level) from "what language is being used" (utterance-level). This prevents the acoustic features from being "taxed" with the burden of language identification.
  • The "Skip Convolution" Trick: Allowing the Summary Vector to skip convolutions is vital. Convolutions extract local patterns; by skipping them, the vector maintains a "pure" global view of the entire audio clip.
  • Robustness to Uncertainty: The model is trained with "multihot" prompts (e.g., "this audio could be Spanish or Italian"). This mimics real-world scenarios where the exact language might be unknown, but the geographical context is restricted.
  • Saturation Point: Adding adapters to every single layer isn't always beneficial. Performance degraded when adapters were added to the final layers (e.g., the 12th layer), suggesting these layers should focus purely on text decoding.

Key Results

The csvMASR model demonstrated superior performance across multiple metrics compared to standard multilingual baselines.

Comparison of WER across different models and LID prompting strategies on the MLS dataset.
  • Accuracy: csvMASR-4 reduced the aggregate Word Error Rate (WER) to 9.95%, compared to the multilingual baseline of 10.33%.
  • Language ID Mastery: In language classification tasks, csvMASR achieved up to 16.65% higher accuracy than "Framewise" weighted interpolation models.
  • Configurability: The model showed a very narrow WER gap (less than 1%) between "1-hot" prompts (exact language known) and "all-hot" prompts (no prior knowledge). This proves the model is excellent at automatically adapting to the correct language even without a specific prompt.
  • Prompt Sensitivity: Unlike the LIDConcat method, which often ignores user prompts, csvMASR's performance changes predictably when different language prompts are provided, demonstrating true configurability.

Conclusion and Future Work

While csvMASR shows significant promise on seven languages, future work will involve testing its scalability to dozens or hundreds of languages. Additionally, integrating high-resource languages like English without biasing the adapters remains a challenge. Finally, while parameter-efficient, the routing logic adds complexity that could be further optimized for real-time streaming applications.

Summary updated

BibTeX
@misc{zhu2024configurable,
  author = {Harrison Zhu and Ivan Fung and Yingke Zhu and Lahiru Samarakoon},
  title = {{Configurable Multilingual ASR with Speech Summary Representations}},
  year = {2024},
  eprint = {2410.04478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.04478},
}
Figure