Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter

Song Li, Yongbin You, Xuezhi Wang, Ke Ding, Guanglu Wan

TL;DR

Researchers from Meituan have developed two parameter-efficient methods to boost multilingual speech recognition: Language Prompt Tuning for when the language is known beforehand, and a Frame-Level Language Adapter for when it isn't. By combining these techniques, they achieved significant accuracy gains across seven languages—including low-resource ones—while keeping the model architecture unified and efficient.

What Problem Does This Paper Address?

Building a single "all-in-one" speech recognition model that handles multiple languages is a double-edged sword. While it helps low-resource languages by sharing knowledge from high-resource ones, it often leads to "language confusion." This happens when the model struggles to distinguish between similar sounds in different languages, causing performance to drop for major languages like English or Chinese.

The authors identify two distinct real-world scenarios:

  1. Language-Configurable: The user manually selects the language (e.g., a "switch" in an app).
  2. Language-Agnostic: The system must automatically detect and transcribe the language without prior knowledge.

Most previous research treats these as separate problems. This paper aims to solve both simultaneously using a single, parameter-efficient framework.

Methods

The authors propose two main architectural "add-ons" for a standard Transformer-CTC speech model:

1. Language Prompt Tuning (Configurable Scenario)

Inspired by how Large Language Models (LLMs) use prompts, the authors introduce a Prompt Encoder. When the language is known, a discrete Language ID (LID) is converted into a continuous embedding (a "prompt").

  • The "Aha" Moment: They found that appending this prompt as a suffix to the acoustic features worked better than a prefix. This ensures the model processes the raw acoustic signal first without breaking its continuity, while still using the prompt to "bias" the final recognition.

2. Frame-Level Language Adapter (Agnostic Scenario)

For cases where the language is unknown, the authors built a Frame-Level Language Adapter (FL-Adapter).

  • Fine-grained Bias: Unlike traditional methods that predict the language once for the whole sentence, this module predicts the language for every single frame of audio.
  • CTC Loss for LID: They used Connectionist Temporal Classification (CTC) loss to train the language identification. This allows the model to naturally handle non-speech segments (like silence or noise) by aligning them to a "blank" label rather than forcing a language prediction where none exists.

3. Unified Model

To create a single system that handles both scenarios, they used parameter-efficient fine-tuning. They froze the main model and only trained the small prompt and adapter modules. This allows the system to be highly flexible without needing to retrain a massive model from scratch.

Datasets

The researchers tested their methods on a diverse set of seven languages totaling 8,723 hours of audio:

  • High-Resource: Chinese (6,200h), English (961h), Japanese (930h), and Spanish (435h).
  • Low-Resource: Tibetan (84.3h), Korean (51.6h), and Uyghur (21h).

They used well-known open-source sets like AISHELL-1, Librispeech, and Common Voice. To handle different scripts (like Chinese characters vs. the Latin alphabet), they used Byte Pair Encoding (BPE) to create a shared vocabulary of 6,000 units.

Key Insights and Ideas

  • Continuity Matters: In prompt tuning, placing the language hint at the end (suffix) is superior to the beginning (prefix). This suggests that maintaining the integrity of the initial acoustic features is vital for the Transformer's self-attention mechanism.
  • Silence is Golden: Using CTC loss for language identification is a clever trick. Because it includes a "blank" symbol, the model doesn't get "confused" by silence or background noise, leading to much cleaner language bias.
  • Efficiency Wins: The proposed methods add very few parameters (the "Params" column in their results shows only minor increases), yet they outperform much larger, more complex baseline models.

Key Results

The improvements were substantial across the board. Using the Word Error Rate (WER)—where lower is better:

  • Baseline: 26.05% average WER.
  • Configurable Model (Prompt Tuning): Dropped to 18.38%.
  • Agnostic Model (FL-Adapter): Dropped to 19.64%.
  • Unified Model: Achieved a final average WER of 18.37%.

The most dramatic gains were seen in low-resource languages. For example, Uyghur WER dropped from a staggering 83.18% in the baseline to 54.09% in the unified model.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Scaling: The current study used seven languages. Future work will test if these "prompts" and "adapters" hold up when scaled to dozens or hundreds of languages.
  • Model Size: The experiments were conducted on a medium-sized Transformer. The authors plan to validate these methods on much larger "foundation" models.
  • Real-time Constraints: While parameter-efficient, the frame-level adapter adds some computational overhead that needs to be optimized for edge devices (like smartphones).

Summary updated

Abstract

Multilingual intelligent assistants, such as ChatGPT, have recently gained popularity. To further expand the applications of multilingual artificial intelligence (AI) assistants and facilitate international communication, it is essential to enhance the performance of multilingual speech recognition, which is a crucial component of speech interaction. In this paper, we propose two simple and parameter-efficient methods: language prompt tuning and frame-level language adapter, to respectively enhance language-configurable and language-agnostic multilingual speech recognition. Additionally, we explore the feasibility of integrating these two approaches using parameter-efficient fine-tuning methods. Our experiments demonstrate significant performance improvements across seven languages using our proposed methods.
BibTeX
@inproceedings{li2024enhancing,
  author = {Song Li and Yongbin You and Xuezhi Wang and Ke Ding and Guanglu Wan},
  title = {{Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=7stIPenfgU},
}

Loading related content...

Figure