Abstract
Multilingual intelligent assistants, such as ChatGPT, have recently gained popularity. To further expand the applications of multilingual artificial intelligence assistants and facilitate international communication, it is essential to enhance the performance of multilingual speech recognition, which is a crucial component of speech interaction. In this paper, we propose two simple and parameter-efficient methods: language prompt tuning and frame-level language adapter, to respectively enhance language-configurable and language-agnostic multilingual speech recognition. Additionally, we explore the feasibility of integrating these two approaches using parameter-efficient fine-tuning methods. Our experiments demonstrate significant performance improvements across seven languages using our proposed methods.
Introduction
Building a single "all-in-one" speech recognition model for multiple languages presents a significant challenge: "language confusion." While sharing knowledge across languages benefits low-resource ones, it can degrade performance for high-resource languages like English or Chinese as the model struggles to distinguish between phonetically similar sounds.
This research addresses two distinct real-world scenarios:
- Language-Configurable: The user manually selects the language (e.g., a setting in an app).
- Language-Agnostic: The system must automatically detect and transcribe the language without prior knowledge.
Methodology
The authors propose two architectural enhancements for a standard Transformer-CTC speech model to address these scenarios efficiently.
Language Prompt Tuning (Configurable Scenario)
For scenarios where the language is pre-selected, a Prompt Encoder converts a discrete Language ID (LID) into a continuous embedding (a "prompt"). The researchers explored several ways to integrate these prompts, including concatenation, addition, and attention-based methods.
A key finding was that appending the prompt as a suffix to acoustic features outperformed prefixing. This ensures the model processes the raw acoustic signal first, maintaining its continuity for the Transformer's self-attention mechanism, while still using the prompt to bias the final recognition.
Frame-Level Language Adapter (Agnostic Scenario)
In language-agnostic settings, the Frame-Level Language Adapter (FL-Adapter) provides fine-grained language bias. Unlike traditional sentence-level identification, this module predicts the language for every single frame of audio.
By utilizing Connectionist Temporal Classification (CTC) loss for language identification, the model effectively handles non-speech segments (like silence or noise) by aligning them with a "blank" label. This prevents the model from forcing a language prediction where none exists, leading to a cleaner language bias.
Unified Model Integration
To create a single system that handles both scenarios, the authors employed parameter-efficient fine-tuning. By freezing the backbone model and training only the small prompt and adapter modules, they created a flexible system that achieves high performance without the computational cost of full model retraining.
Experimental Setup
The methods were evaluated on a diverse dataset totaling 8,723 hours across seven languages:
- High-Resource: Chinese (6,200h), English (961h), Japanese (930h), and Spanish (435h).
- Low-Resource: Tibetan (84.3h), Korean (51.6h), and Uyghur (21h).
A shared vocabulary of 6,000 Byte Pair Encoding (BPE) units was used to accommodate the various scripts and phonetic structures across these languages.
Results and Discussion
The proposed methods yielded substantial improvements in Word Error Rate (WER), where lower values indicate better performance:
- Baseline: 26.05% average WER.
- Configurable Model (Prompt Tuning): Reduced to 18.38%.
- Agnostic Model (FL-Adapter): Reduced to 19.64%.
- Unified Model: Achieved the best overall performance at 18.37%.
The most dramatic gains were observed in low-resource languages. For instance, the WER for Uyghur dropped from 83.18% in the baseline to 54.09% in the unified model, demonstrating the effectiveness of cross-lingual knowledge sharing and targeted language biasing.
Conclusion
This work demonstrates that language prompt tuning and frame-level adapters are effective, parameter-efficient strategies for enhancing multilingual ASR. By addressing both configurable and agnostic scenarios within a unified framework, the researchers have provided a scalable path for improving multilingual intelligent assistants. Future work will focus on scaling these methods to hundreds of languages and validating them on larger foundation models.
Summary updated