TL;DR
This paper introduces a novel method for generating word lattices in large-vocabulary continuous speech recognition (LVCSR) systems. By combining acoustic-based word candidates with traditional language model-driven lattices, the authors achieve improved accuracy and robustness against out-of-grammar sentences and error propagation.
What Problem Does This Paper Address?
Modern speech recognition systems, especially those dealing with large vocabularies and continuous speech (LVCSR), heavily rely on language models to constrain the search space and improve accuracy. However, this reliance can lead to two significant problems:
- Out-of-Grammar (OOG) Sentences: Language models struggle to predict sentences that deviate from their learned patterns, leading to recognition errors. This is particularly problematic in applications like voice search or dialogue systems where users might utter unexpected phrases.
- Error Propagation: When a word is misrecognized, the language model's strong dependency can cause subsequent words to be misrecognized as well, creating a cascade of errors.
The paper aims to mitigate these issues by making the word lattice generation process more robust to acoustic evidence, even when it doesn't perfectly align with the language model.
Methods
The core contribution is a hybrid word lattice generation approach. This method augments the standard LVCSR lattice generation process with an acoustic-based word candidate detection module.
The process involves two main stages:
- Individual Word Candidates:
- Phone Recognition: The input speech is first converted into a phonetic sequence.
- Word Matching: This phonetic sequence is then matched against the phonetic representations of words in the vocabulary.
- Rescoring: Promising word candidates are rescored using more complex acoustic models and refined endpoint detection (using iterative Viterbi decoding) to improve acoustic score accuracy.
- Candidate Sequences:
- Length-Adjusted Scoring: To address the issue of short, frequently occurring words being favored over longer, more meaningful ones, candidates are scored based on their acoustic likelihood per frame.
- Sequence Scoring: To further improve robustness, frequently occurring word sequences are also considered, combining acoustic scores with n-gram language model scores.
The final hybrid lattice is constructed by combining the word candidates detected by this acoustic-focused module with the lattice generated by the traditional CSR system. This aims to preserve acoustically strong but linguistically less probable words that might otherwise be pruned.
Datasets
- Training Data: Approximately 146 hours of Korean speech data sampled at 16kHz. A trigram language model was trained on 3 million words from the Sejong DB.
- Test Data: Two sets of fifty distinct sentences each.
- BASIC style: Sentences guaranteed to be predictable by the n-gram language model.
- FREE style: Paraphrased versions of BASIC sentences, designed to be more challenging and closer to real-world utterances.
- Microphones: Utterances were recorded using both a headset microphone (Sennheiser ME3) and a microphone array (Acoustic Magic Voice Tracker) at a distance.
- Corrupted Signal: Test utterances were artificially corrupted by superimposing white noise (-20dB) over 10-30% of the signal length, positioned in the first half, to simulate misrecognition and error propagation.
Key Insights and Ideas
- Acoustic Evidence is Crucial: The paper strongly advocates for giving more weight to acoustic evidence, especially for word candidates that might be overlooked by a purely language-model-driven approach.
- Hybridization as a Solution: Combining the strengths of acoustic-based detection (robustness to OOG) and language-model-based lattices (efficiency, OOV handling) offers a synergistic improvement.
- Beyond Simple Pruning: The hybrid approach is presented as a more effective way to handle OOG and error propagation than simply reducing pruning in the traditional CSR lattice generation.
- Trade-offs in Lattice Density: The study highlights that improved lattice accuracy can be achieved even with similar or slightly lower lattice density, suggesting efficiency gains are possible.
Key Results
- Lattice Accuracy Improvement: The hybrid lattice achieved a 33% improvement in lattice accuracy under similar lattice density conditions.
- Improved Word and Sentence Accuracy: Table 1 shows significant increases in both word and sentence accuracies across different evaluation sets (e.g., BASIC, FREE style, NEAR/FAR microphone distances). For instance, the FREE style with a FAR microphone saw a 56.7% improvement in word error rate.
- Robustness to OOG: The FREE style sentences, which are more likely to contain OOG elements, showed a higher improvement, indicating the hybrid method's effectiveness in handling such cases.
- Reduced Error Propagation: Experiments with corrupted signals (Table 3) demonstrated that the hybrid system is less susceptible to error propagation compared to the baseline CSR system. The corruption rate (words not detected in corrupted vs. clean signal) was lower for the hybrid model.
Limitations and Future Work
- Limitations:
- The paper notes that the improvement in the distant microphone case is less obvious, suggesting the acoustic model's accuracy plays a larger role there.
- While the hybrid method compensates for OOG to a certain extent, it doesn't entirely eliminate the problem.
- Future Work:
- The authors suggest that further work is required in building a complete CSR system that effectively leverages this hybrid word lattice.
Summary updated