TL;DR
This paper introduces a novel method for automatically identifying and filtering recognition errors in speech transcripts by detecting "semantic outliers." This approach significantly reduces errors in content words, making audio documents easier to browse and understand, especially for noisy or less-than-ideal audio recordings.
What Problem Does This Paper Address?
As spoken audio documents become more prevalent (think conference calls, video recordings), efficiently browsing and "gisting" them is a growing challenge. While automatic speech recognition (ASR) can generate transcripts, these often contain numerous errors, making them distracting and hindering comprehension. The paper tackles the problem of how to automatically identify and remove these recognition errors from transcripts to improve their usability for tasks like browsing and gisting, without requiring real-time processing.
Methods
The core of the paper's method is to identify "semantic outliers" – words that don't cohere well with their surrounding words in the transcript. The assumption is that these outliers are often mistranscribed words. The algorithm works as follows:
- Neighborhood Definition: For each content word (non-stopwords), a "neighborhood" of surrounding content words is defined. Two variants were explored: using all content words in the transcript ("All" variant) or a fixed window of words before and after the target word ("Window" variant).
- Pair-wise Semantic Similarity: The semantic similarity between all pairs of words within a neighborhood is computed. The paper experimented with two measures:
- Roget variant: A dictionary-based measure using Roget's thesaurus.
- PMI variant: A corpus-based measure using Pointwise Mutual Information (PMI) calculated on a large web corpus.
- Semantic Coherence Calculation: For each word in the neighborhood, its "semantic coherence" is calculated by aggregating its pair-wise similarities with other words in the neighborhood. Three aggregation schemes were tested:
- AVG: Averaging all pair-wise similarities.
- MAX: Taking the maximum similarity to any neighbor.
- 3MAX: Averaging the three most similar neighbors.
- Error Detection: A word is flagged as a recognition error if its semantic coherence is below a certain threshold (K) relative to the average semantic coherence of its neighbors. The parameter K controls the aggressiveness of the filtering.
The paper explores 12 configurations of this algorithm (2 neighborhood variants x 2 similarity measures x 3 aggregation schemes).
Datasets
The experiments were conducted on a randomly selected subset of 100 stories from the TDT2 English Audio corpus. Two types of ASR transcripts were used:
- BBN dataset: Generated by the NIST/BBN time-adaptive speech recognizer, with a moderate Word Error Rate (WER) of 27.6%. This is representative of speaker-independent ASR systems tuned for broadcast news.
- Dragon dataset: Generated using the Dragon NaturallySpeaking speaker-dependent recognizer, with a much higher WER of 62.3%. This approximates the type of high WER seen in casual, less-than-broadcast quality audio.
Key Insights and Ideas
- Semantic Coherence as an Error Indicator: The fundamental idea that words semantically "out of place" are likely transcription errors is a powerful heuristic.
- PMI Outperforms Roget for Semantic Similarity: For the task of detecting ASR errors, the corpus-based PMI measure showed significantly better performance than the dictionary-based Roget measure, especially on noisier transcripts. This highlights the advantage of leveraging large-scale web data for semantic relatedness.
- Windowing Can Be Beneficial, Especially for High WER: While the "All" variant (using the entire transcript) was explored, the "Window" variant (using a local context) showed marginal improvements, particularly for the higher WER Dragon dataset, suggesting that local context is crucial for error detection.
- Precision is Key for Browsing: For audio browsing and gisting, the paper argues that Precision (correctly identifying errors without removing too many good words) is more important than Recall (finding all errors). This led to favoring the "3MAX" aggregation scheme.
- Trade-off Between Error Reduction and Lost Content: The filtering approach effectively reduces transcription errors but inevitably leads to the loss of some correctly transcribed content words. This is a crucial trade-off to consider.
Key Results
- Significant Error Reduction: The best performing configuration (Window-PMI-3MAX) was able to significantly reduce the Content Word Error Rate (cWER). For example, on the moderately accurate BBN dataset, cWER could be reduced by 50% while losing 45% of good content words (at K=100). For the high-accuracy Dragon dataset, a 50% cWER reduction was achieved with a 50% loss of good content words (at K=120).
- PMI Superiority: Figures 1 and 2 clearly show that the PMI-based similarity measure consistently outperforms the Roget-based measure across different recall levels for both datasets.
- No WER Improvement (Overall): The paper notes that overall Word Error Rate (WER) did not improve, primarily due to the penalty incurred by deletions (i.e., the filtered-out words). However, they argue that for gisting and browsing, reducing errors even at the cost of some deletions is preferable.
- Comparison to Prior Work: The paper's PMI-based approach appears to perform better than the LSA-based measure reported by Cox and Dasmahapatra (2000) in terms of Precision-Recall curves, particularly at higher precision levels.
Limitations and Future Work
- Loss of Good Content Words: A significant limitation is the percentage of correctly transcribed content words that are lost during the filtering process. While acceptable for gisting, this is a trade-off.
- Computational Intensity: The semantic similarity computation (Step 2) is computationally intensive, though this is acceptable as real-time correction is not required.
- Future Work:
- Experimenting with multi-topic audio documents and exploring alternative aggregation schemes (MAX, 3MAX) more thoroughly.
- Scaling up corpus-based semantic similarity measures to even larger corpora.
- Exploring other outlier detection methods like clustering or lexical chains.
- Combining their method with internal ASR confidence measures to improve both precision and recall.
- Investigating the identification and retention of named entities.
- Developing methods to correct errors rather than just filter them out, potentially by selecting the best hypothesis from N-best lists based on semantic cohesion.
- Conducting human subject studies to evaluate the practical benefits of filtered transcripts for browsing and gisting tasks.
Summary updated