Abstract
Due to the dynamic nature of human language, automatic speech recognition (ASR) systems must continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending terms and new named entities, pose significant challenges to modern end-to-end (E2E) ASR systems, which often require extensive training to adapt their large parameter sets. Unlike previous research focused on language model post-processing, this paper addresses the problem at the acoustic modeling level to eliminate bias against OOV words. We propose generating OOV training data using text-to-speech (TTS) systems and applying loss rescaling to encourage neural networks to prioritize these unseen words. Specifically, we explore enlarging classification loss at the sentence level and rescaling gradients at the word level during fine-tuning. To prevent catastrophic forgetting, we combine loss rescaling with model regularization techniques like Elastic Weight Consolidation (EWC). Experimental results on the LibriSpeech benchmark demonstrate that our word-level rescaling approach significantly improves OOV recall rates with minimal impact on overall word error rate (WER).
Introduction
Modern end-to-end (E2E) Automatic Speech Recognition (ASR) systems have revolutionized the field, yet they remain inherently biased toward the vocabulary present in their training data. When new words—such as "COVID-19," new brand names, or technical jargon—enter common parlance, these "black box" models struggle to recognize them. Because E2E models lack a modular, easily updatable lexicon, they often require retraining or complex post-processing to adapt.
Traditionally, researchers have attempted to solve the OOV problem using external language models during post-processing. However, if the acoustic model fails to generate the correct phonetic or character-based representation initially, linguistic post-processing is often insufficient. This paper argues for a shift in focus: eliminating bias at the acoustic level. By using synthetic audio and specialized loss functions, we can teach models to "hear" and recognize new words without the prohibitive cost of collecting real-world audio for every new term.
Methodology
1. Synthetic Data Generation
To bridge the data gap, we identify 100 target OOV words and extract sentences containing them from web crawls. We then utilize multiple commercial Text-to-Speech (TTS) APIs (including Google and Baidu) to generate diverse synthetic audio. This ensures the model is exposed to various synthetic voices and prosodic patterns, providing a robust training set for the new vocabulary.
2. Loss Rescaling
The core of our approach is "loss rescaling," which forces the model to prioritize gradients from OOV words during the fine-tuning process. We evaluated two primary methods:
- Sentence-level Rescaling: The loss for an entire utterance is multiplied by a hyper-parameter ($\mu$) if it contains an OOV word. While simple, this can lead to gradient explosion and unintended noise amplification.
- Word-level Rescaling: A more precise method that modifies the Connectionist Temporal Classification (CTC) loss. By specifically increasing the gradients for nodes in the decoding lattice corresponding to OOV subwords, the model learns to focus on the specific acoustic segments of interest. We also found that enhancing the gradients of "blank" tokens immediately surrounding OOV words is vital for learning word boundaries.
3. Preventing Catastrophic Forgetting
Fine-tuning on a small set of synthetic OOV data often causes the model to forget its original training (e.g., LibriSpeech). To mitigate this, we employ Elastic Weight Consolidation (EWC). EWC acts as a regularizer that penalizes changes to parameters deemed important for the original task, allowing the model to learn new words using only its less critical weights.
Experimental Setup and Results
Datasets
We pre-trained our baseline model on the LibriSpeech (960h) dataset. For OOV testing, we used LRS3-TED to identify words not present in LibriSpeech, creating a Real OOV (ROOV) test set. Our training was supplemented by a Synthetic OOV (SOOV) dataset featuring 8 different speaker voices across 5 TTS APIs.
Performance Metrics
The results indicate a dramatic improvement in the model's ability to recognize new terms:
- Recall Rate: Using word-level loss rescaling combined with EWC, the model achieved a 45.81% recall rate on real-world OOV words, compared to a baseline of just 1.37%.
- Stability: Word-level rescaling proved more stable than sentence-level rescaling, offering higher precision and avoiding the "gradient explosion" issues associated with boosting entire utterances.
- Minimal Degradation: The Word Error Rate (WER) on the standard LibriSpeech "test-clean" set increased by only a relative 7.8%. This demonstrates that EWC successfully preserved the model's core performance while expanding its vocabulary.
Conclusion
This research demonstrates that acoustic-level bias is a primary hurdle in OOV recognition for E2E ASR systems. By combining synthetic data generation with word-level loss rescaling and EWC regularization, we can effectively teach models new vocabulary with high recall and minimal performance loss on existing tasks. Future work will focus on scaling this method to thousands of OOV words and adapting the technique for more complex architectures like RNN-T and pure attention-based models.
Summary updated