TL;DR
This paper introduces a novel unsupervised approach to detect and correct errors in automatic speech recognition (ASR) output, specifically focusing on misrecognized query words. By analyzing word co-occurrence patterns in ASR output and leveraging phonetic similarity, the method aims to improve information retrieval and other downstream tasks that rely on accurate speech transcripts.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are increasingly used to transcribe spoken language for various applications like information retrieval, question answering, and summarization. However, ASR systems are prone to errors, often misrecognizing words. These errors can significantly degrade the performance of downstream systems. For instance, if a user searches for documents related to "Iraq" and the ASR system consistently misrecognizes this word, the user will fail to find relevant documents. This paper addresses the challenge of identifying and correcting these misrecognized words, particularly those that are part of a user's query, within the output of an ASR system. The focus is on improving the accuracy of specific, queried terms rather than reducing the overall word error rate of the ASR system itself.
Methods
The proposed approach is a three-step, unsupervised post-processing technique that operates independently of the ASR engine:
- Word Co-occurrence Analysis: This offline step involves analyzing a large corpus of ASR output to build a statistical model of word contexts. For each word in the ASR vocabulary, the system identifies other words that frequently co-occur within a defined window. This is achieved by calculating pointwise mutual information (PMI) between words, which highlights words that appear together more often than expected by chance. This analysis generates ranked lists of context words for each vocabulary word.
- Context Analysis: In this step, the system identifies "contextual regions" within a test document that are likely to contain a specific query word or its misrecognition. This is done by sliding a window across the document and checking if a sufficient number of the query word's expected context words (identified in step 1) are present within that window. Words in such windows are flagged as potential candidate errors.
- Phonetic Comparison: For each candidate error identified in step 2, a phonetic similarity check is performed against the original query word. This uses a weighted phonetic alignment technique to measure the distance between the pronunciations of the candidate word and the query word, based on the ASR system's lexicon. A strong phonetic similarity, combined with the contextual evidence, increases the confidence that the candidate word is indeed a misrecognition of the query word.
The paper also mentions a probabilistic pruning step using Kullback-Leibler divergence to further refine candidate errors by comparing the probability of observing the context words with the query word versus the candidate word.
Datasets
The authors used a large corpus of English broadcast news transcripts produced by the broadcast news speech system described in Makhoul et al. (2000).
- Training Corpus: 360 half-hour broadcast transcripts, containing approximately 1.6 million words. These broadcasts were from CNN Headline News, BBC America, and News World International from July 2003.
- Evaluation Corpus: 39 half-hour broadcast transcripts, containing about 180,000 words. This corpus was used for testing and was randomly selected from the larger corpus.
The paper does not introduce new benchmarks but rather evaluates its approach on existing ASR output data.
Key Insights and Ideas
- Leveraging ASR Errors for Context Modeling: A key insight is that systematic and recurring ASR errors themselves can provide valuable contextual information. For example, the misrecognition "inge" for "in going" in the context "in going to war with Iraq" can be informative. This contrasts with approaches that rely solely on clean transcripts.
- Unsupervised Approach: The entire method is unsupervised, meaning it doesn't require labeled data for training the error detection and correction models, making it scalable to large datasets.
- Complementary to ASR Improvement: The approach is designed as a post-processing step, meaning it can be applied to the output of any ASR system and can adapt to incremental improvements in ASR technology.
- Focus on Query Words: The paper highlights the importance of accurately recognizing query words for user-centered tasks like information retrieval, arguing that improving specific query word accuracy can have a significant impact even if the overall word error rate isn't drastically reduced.
Key Results
The initial evaluation focused on three query words: "Iraq," "Abbas," and "Lynch."
- "Iraq": This proved to be the most fruitful query due to its prevalence. Out of 142 misrecognized instances, the system achieved high precision and recall for certain window sizes and minimum context word thresholds. For a window size of 14 and minimum context words of 2, recall was 95% and precision was 98%. Table 2 shows that for a window size of 14, increasing the minimum context words from 1 to 5 led to a decrease in detected errors but a perfect precision of 100% for 3, 4, and 5 context words.
- "Abbas": In the test corpus, 10 examples of "Abbas" were misrecognized. The method detected 8 candidates, 7 of which were actual misrecognitions, resulting in a recall of 70% and a precision of 88% (with window size w=10, minimum context c=2).
- "Lynch": This query word was misrecognized only 4 times, primarily as "lynched." The context matching worked well, identifying relevant context words.
The paper emphasizes that while these results are preliminary, they demonstrate the potential for achieving high precision with reasonable recall.
Limitations and Future Work
- Preliminary Evaluation: The authors acknowledge that the results are preliminary and that a broader evaluation with more query words and test cases is needed to assess the general applicability of the approach.
- Impact on Retrieval Performance: While the authors believe there's an obvious benefit to correcting misrecognized query terms, they have not formally evaluated the impact of their error detection and correction on retrieval performance. This is a clear direction for future work.
- Modeling False Positives: The use of ASR output without clean transcripts for training introduces the possibility of modeling false positives, which the authors note as a potential issue.
- General Applicability: The current evaluation is limited to broadcast news data. Further testing on other types of spoken language data (e.g., meetings, telephone conversations) would be beneficial.
Summary updated