TL;DR
This paper introduces SeACo-Paraformer, a novel non-autoregressive ASR system designed for highly flexible and effective hotword customization. By integrating a semantic-augmented contextualization module with the efficient Paraformer backbone, SeACo-Paraformer significantly improves the accuracy and recall of personalized phrases without sacrificing general ASR performance, even with large hotword lists.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are becoming increasingly ubiquitous, but a key challenge remains: enabling users to customize the system to recognize specific words or phrases, such as personal names, product names, or domain-specific terminology. This "hotword customization" is crucial for improving user experience, especially in commercial applications. While previous methods like CLAS (Contextual Listen, Attend, and Spell) and its extensions have made progress, they often struggle with training stability, the "invisible" nature of their customization process, and performance degradation when dealing with a large number of hotwords. Existing approaches also sometimes fail to decouple general ASR modeling from contextualization, leading to issues when encountering challenging cases. This paper aims to address these limitations by proposing a more robust, flexible, and efficient hotword customization solution.
Methods
The core of this work is the SeACo-Paraformer (Semantic-Augmented Contextual-Paraformer), built upon the Paraformer non-autoregressive ASR model. Paraformer itself is known for its efficiency and accuracy, leveraging a Continuous Integrate-and-Fire (CIF) mechanism for non-autoregressive decoding.
SeACo-Paraformer introduces three key modules:
- CIF Predictor for Hotword Prediction: The CIF predictor, originally used for acoustic embedding generation in Paraformer, is repurposed to also predict hotword information.
- Bias Decoder: This module takes the CIF output and the Paraformer's parallel decoder hidden states and injects hotword information through multi-headed attention mechanisms. This allows the system to learn relationships between acoustic/semantic representations and the target hotwords.
- Attention Score Filtering (ASF): To combat the performance drop associated with large hotword lists, ASF is introduced. It filters the incoming hotword list by analyzing attention scores from the bias decoder, selecting the most relevant hotwords for more effective customization.
The training process for the hotword customization modules is decoupled from the main ASR training, allowing for the use of specialized hotword data and strategies without negatively impacting general ASR performance. Inference involves merging the probabilities from the standard ASR output and the biased output from the SeACo-Paraformer, with a tunable parameter to balance their influence.
Datasets
The paper utilizes a large-scale industrial dataset of approximately 50,000 hours for training. For evaluation, they employ several test sets:
- Test-General: A broad collection of general ASR test sets.
- Test-Commercial, Test-Term, Test-Entity: Domain-specific hotword test sets.
- Test-Aishell1-NE and Dev-Aishell1-NE: Newly constructed benchmark datasets derived from the open-source Aishell-1 dataset. These were created by extracting named entities from transcriptions and using them as hotwords, with a focus on "R1-hotwords" (those with a recall rate below 40% in the base ASR model) to specifically test customization capabilities.
Key Insights and Ideas
- Decoupled Training for Customization: Training the hotword customization modules separately from the main ASR backbone is a critical design choice. It allows for specialization without compromising general ASR accuracy, a significant advantage over methods that might conflate these two aspects.
- CIF's Dual Role: Leveraging the CIF predictor, already present in Paraformer, for both acoustic embedding generation and hotword prediction is an elegant way to integrate customization without adding excessive architectural complexity.
- ASF's Practical Necessity: The observation that customization performance degrades with increasing hotword list size is a practical challenge. ASF directly addresses this by intelligently filtering hotwords, demonstrating that not all hotwords are equally important for a given utterance.
- Attention Score as a Confidence Measure: The cumulative attention scores across all steps for each hotword serve as a proxy for their likelihood of occurrence, forming the basis of the ASF strategy. This provides a quantifiable way to assess hotword relevance.
- Bias Decoder's Selective Attention: The analysis of the bias decoder's attention score matrix reveals that semantic information selectively attends to non-blank phrases only when relevant hotwords are present, indicating a sophisticated mechanism for contextualization.
Key Results
- SeACo-Paraformer achieved a 5% relative CER reduction compared to Paraformer-CLAS on industrial test sets.
- On open-source test sets, SeACo-Paraformer and SeACo-Paraformer+ASF achieved 79% and 87% recall rates, respectively, outperforming Paraformer-CLAS.
- The proposed method achieved a 30.2% CER reduction and 26.1% recall improvement compared to Paraformer-CLAS on these open-source sets.
- With the ASF strategy, the recall rate for hotwords increased to 65% on internal tests.
- Experiments with increasing hotword numbers showed that SeACo-Paraformer+ASF maintained its performance advantage, even achieving a CER reduction when the hotword list expanded to 4000.
- Ablation studies confirmed that using both acoustic embeddings (CIF output) and decoder hidden states is crucial for stable hotword prediction, and merging them after multi-headed attention (rather than before) yielded better results.
Limitations and Future Work
- Performance Degradation with Large Lists: While ASF mitigates the issue, the paper acknowledges that performance still deteriorates as the hotword list grows significantly.
- Future Work: The authors plan to focus on handling even larger-scale incoming hotword lists and exploring different bias encoder structures. They also aim to further investigate the fundamental concept of ASF and its potential applications.
Summary updated