TL;DR
This paper explores two methods for adapting language models to new domains: Maximum A Posteriori (MAP) estimation and the perceptron algorithm. They find that combining these approaches, particularly by using the perceptron algorithm on early recognition pass lattices, yields a small but significant improvement in word error rate, demonstrating the value of a hybrid strategy.
What Problem Does This Paper Address?
Speech recognition systems often struggle when the language spoken in a new domain (e.g., customer service calls) differs significantly from the language they were initially trained on (general voicemail). This "domain shift" leads to increased errors. Language model adaptation aims to bridge this gap by adjusting the language model to better reflect the new domain's vocabulary and phrasing. Traditional methods like MAP estimation adjust model parameters to match the new domain's data. However, this paper investigates a discriminative approach, the perceptron algorithm, which directly tries to correct recognition errors. The core challenge is to find the most effective way to leverage these different adaptation techniques, especially in a multi-pass recognition system where the output of one pass informs the next.
Methods
The paper compares two primary language model adaptation techniques:
- MAP Estimation: This approach starts with a general language model and "merges" counts from a new, in-domain corpus. It's essentially a smoothed version of maximum likelihood estimation, where the degree of reliance on the original model is controlled by a mixing parameter. The goal is to adjust the model parameters to match the in-domain distribution.
- Perceptron Algorithm: This is a discriminative training method. It uses a baseline speech recognizer to generate word lattices (hypotheses of possible transcriptions) for in-domain utterances. The perceptron algorithm then adjusts its model parameters to minimize the word error rate (WER) on these lattices, specifically by favoring the gold-standard transcription over other hypotheses. A key aspect is that the perceptron is trained on lattices produced by a recognizer that doesn't see the gold-standard transcription during its initial pass, a technique shown to be crucial for improvements.
The paper also explores combining these methods within a multi-pass recognition system. This involves using the output of one recognition pass (e.g., a word lattice) to train or adapt the model for a subsequent pass.
Datasets
- Training Data: Approximately 100 hours of general voicemail messages (used for the baseline Scanmail system).
- Adaptation Data: 17 hours of manually transcribed customer service voicemail messages (named SS-NIFR).
- Test Data: 1 hour of manually transcribed customer service voicemail messages (SS-NIFR).
The paper does not introduce new benchmark datasets but uses a specific customer service domain (SS-NIFR) to evaluate adaptation.
Key Insights and Ideas
- MAP vs. Perceptron in Isolation: MAP estimation, when used alone, generally outperforms the perceptron algorithm on its own. This is attributed to MAP's ability to consider all possible hypotheses in the initial search pass, whereas the perceptron is constrained by the hypotheses present in the lattice generated by the unadapted model.
- The Power of Combination: Combining MAP estimation and the perceptron algorithm yields better results than either method alone. This highlights the complementary nature of their objectives: MAP provides a good initial model, and the perceptron refines it by directly correcting errors.
- Early Pass Lattices are Key: Using word lattices generated from an earlier recognition pass (e.g., after feature extraction and initial acoustic modeling, denoted FP) for perceptron training leads to better results than using lattices from later, more refined passes (MP). This suggests that cleaner, more accurate transcripts from earlier stages are more beneficial for discriminative adaptation.
- Matching Testing Conditions: It's crucial to match the conditions under which the perceptron model is trained to the conditions under which it's applied during testing. Applying a perceptron trained on MP lattices to FP lattices can actually hurt performance.
Key Results
- The baseline out-of-domain system achieved a Word Error Rate (WER) of 32.7% (FP) and 28.0% (MP) on the SS-NIFR test set.
- MAP estimation alone reduced the WER to 23.7% (FP) and 20.3% (MP).
- The perceptron algorithm alone (trained on FP lattices) achieved 26.8% (FP) and 23.0% (MP).
- The best combined system, using MAP estimation and the perceptron algorithm on early pass lattices, achieved a WER of 19.6% (MP), representing a 0.7% absolute reduction over MAP estimation alone and an 8.4% absolute reduction over the baseline.
Limitations and Future Work
- Perceptron's Lattice Dependency: The perceptron algorithm's performance is limited by the quality and coverage of the hypotheses present in the word lattices generated by the baseline recognizer.
- Computational Cost: While the paper mentions that perceptron training can be made efficient using finite state automata, the multi-pass nature of the combined system implies increased computational cost during recognition.
- Future Work: The paper suggests exploring further combinations of MAP and perceptron within multi-pass systems. They also hint at the importance of closely matching testing conditions for perceptron training, implying that more sophisticated ways to align training and testing might be beneficial. The paper also notes that iterative training (e.g., training a perceptron on lattices produced by a system already using MAP and perceptron) did not yield further improvements in their experiments.
Summary updated