TL;DR
This paper introduces a novel spoken utterance retrieval method that leverages ASR lattices instead of just the single best transcription. This approach significantly improves retrieval accuracy, especially for challenging conversational speech with high word error rates and short audio segments, by better handling recognition uncertainties and out-of-vocabulary words.
What Problem Does This Paper Address?
The core challenge this paper tackles is improving the accuracy of spoken utterance retrieval, particularly for conversational speech like teleconferences. Traditional methods rely on the single best output from Automatic Speech Recognition (ASR) systems to convert speech to text, which is then indexed for retrieval. This works reasonably well for clean, long audio like broadcast news. However, for noisy, short segments with high ASR error rates (up to 50%), this single-best approach leads to poor retrieval performance. The paper aims to bridge this gap by developing a more robust retrieval system that can effectively handle the inherent uncertainties and errors in ASR outputs for conversational speech.
Methods
The paper proposes a lattice-based search approach for spoken utterance retrieval. Instead of using the single best ASR hypothesis, the system utilizes ASR lattices, which represent multiple possible transcriptions along with their associated probabilities.
Key methodological contributions include:
- Lattice Representation: The system indexes and searches directly over word lattices or phone lattices. This allows for a more flexible representation that captures the uncertainty in ASR output.
- Lattice Indexing: A novel indexing scheme is introduced to efficiently store and retrieve information from lattices. This involves creating indices for each arc label (word or phone) and storing information about the lattice number, input/output states, and probabilities.
- Sub-word Unit Indexing (Phone Lattices): To address the problem of Out-Of-Vocabulary (OOV) words, the system also utilizes phone lattices. This involves either direct phone recognition or converting word-level ASR outputs into phone sequences.
- Hybrid Strategies: The paper explores combining word and phone lattices using various strategies, such as simple combination, vocabulary cascade, and search cascade, to leverage the strengths of both.
Datasets
The paper evaluates its methods on three distinct corpora:
- DARPA Broadcast News: A 3-hour evaluation test set from 1998, manually segmented. This serves as a benchmark for cleaner speech.
- Switchboard Corpus: A 5-hour evaluation test set of two-party telephone conversations, manually segmented. This represents a more challenging domain.
- Teleconferences Corpus: A 3.5-hour test set comprising multi-party teleconferences. This is presented as the most challenging domain due to its noisy and conversational nature.
The paper notes that the ASR systems used were trained on large vocabularies (over 200K words for Broadcast News, and over 45K words for Switchboard/Teleconferences).
Key Insights and Ideas
- Lattices are Crucial for Conversational Speech: The most significant insight is that using ASR lattices, rather than just the single best hypothesis, dramatically improves retrieval performance for conversational speech, where ASR errors are high.
- Phone Lattices Help with OOV Words: Incorporating phone lattices is essential for retrieving utterances containing out-of-vocabulary words, which are common in spontaneous speech.
- Hybrid Search is Superior: Combining word and phone lattice searches, particularly using a "search cascade" (word index first, then phone index if no results), yields the best retrieval performance.
- Pruning Lattices Balances Performance and Storage: Pruning word lattices based on cost thresholds offers a way to reduce index size while maintaining good retrieval performance, though it still increases storage compared to single-best hypotheses.
- Minimum Pronunciation Length for Phone Search: For phone-based searches, imposing a minimum pronunciation length for queries helps reduce false alarms without significantly impacting recall.
Key Results
- Significant F-measure Gains: The proposed lattice-based approach improved the maximum F-measure by over 5 points compared to single-best retrieval on tasks with poor WER and low redundancy (e.g., Teleconferences).
- Hybrid System Performance: The final system using both word and phone lattices achieved relative gains of 8-12% in maximum F-measure over the single-best baseline across different tasks.
- Teleconferences Benefit Most: The improvements were most pronounced on the challenging Teleconferences and Switchboard corpora, where ASR performance is poorer. For example, on Teleconferences, the maxF increased from 47.4 (1-best) to 52.8 (word+phone lattices).
- Index Size Increase: Using lattices leads to larger index sizes, ranging from 3x for Broadcast News to 9x for Teleconferences compared to the 1-best baseline.
- Phone Recognition vs. Conversion: Converting word lattices to phone lattices was found to be superior to direct phone recognition for retrieval purposes.
Limitations and Future Work
- Index Size: Acknowledged limitation is the increased index size when using lattices. While pruning helps, it still represents a trade-off.
- Computational Cost: While not explicitly detailed, lattice processing generally implies higher computational costs than single-best hypotheses.
- Future Work: The paper suggests exploring more sophisticated confidence measures and further optimizing hybrid search strategies. The flexibility of the lattice representation also opens avenues for other applications beyond simple retrieval.
Summary updated