TL;DR
This paper tackles the challenge of quickly and accurately identifying the topic of a phone conversation using noisy speech recognition output. They introduce a modified TF-IDF weighting scheme that leverages word confidences and lattice information to significantly improve topic identification accuracy, even when speech recognition is sped up at the cost of accuracy.
What Problem Does This Paper Address?
Organizing and searching large archives of spoken audio is becoming increasingly important. A key step in this process is topic identification – assigning a predefined label (e.g., "sports," "movies") to a conversation. However, performing this task in real-time or on massive datasets requires fast processing, which often means using automatic speech recognition (ASR) systems that are less accurate. The core problem is understanding how ASR errors impact topic identification accuracy and developing techniques to mitigate these degradations, especially when prioritizing speed.
Methods
The authors employ Support Vector Machines (SVMs) for topic identification. Their novel contribution lies in a modified TF-IDF (Term Frequency-Inverse Document Frequency) feature weighting scheme. Instead of simply counting word occurrences, they incorporate:
- Word-level confidences: ASR systems often provide a confidence score for each recognized word. This method discounts words that the ASR system is less certain about.
- Lattice information: ASR systems can output not just the single "best" transcription (1-best) but also a lattice of alternative word sequences with associated probabilities. The paper leverages these posterior probabilities to create a more robust representation.
They define "expected term frequency" (etf) and "expected document frequency" (edf) to probabilistically incorporate these confidence and lattice details into the TF-IDF calculation. This allows them to create feature vectors that are more resilient to ASR errors.
Datasets
- Fisher Corpus: This is the primary dataset used. Specifically, they utilize the English Phase 1 section, employing a 1375-conversation partition for training topic classifiers and a 686-conversation partition for evaluation. Each conversation is treated as a single document with one of 40 predefined topic labels.
- Switchboard Corpus: Used to train the acoustic and language models for the ASR system.
Key Insights and Ideas
- Speed vs. Accuracy Trade-off: The paper demonstrates that while increasing ASR speed leads to higher word error rates (WER), the topic identification performance doesn't degrade proportionally. This suggests that even with more errors, there's still enough signal in the ASR output for topic identification.
- Confidence Matters: Simply treating ASR output as unweighted text is insufficient when errors are present. Incorporating word-level confidences is crucial for robust topic identification.
- Lattices Offer Significant Gains: Using ASR lattices, rather than just the 1-best transcript, provides substantial improvements in topic identification accuracy, especially for faster, less accurate ASR systems. This highlights the value of the richer information available in lattices.
- Probabilistic TF-IDF is Key: The proposed probabilistic TF-IDF calculation (using etf and edf) is essential for effectively leveraging confidence scores and lattice posteriors.
Key Results
- Robustness to Speed: For the fastest ASR system (0.1xRT, ~47% WER), using lattice information with the modified TF-IDF weighting reduced the topic identification error rate by 65% compared to using unweighted 1-best transcripts.
- Comparable Performance: The modified TF-IDF approach, when applied to lattices from a fast ASR system (1xRT), achieved topic identification accuracy comparable to a much slower, more accurate system (10xRT) using unweighted transcripts.
- Baseline: Human-generated transcripts achieved an ID error rate of 10.2%. The best system using ASR output (10xRT with lattices) achieved a similar 10.1% ID error rate.
Limitations and Future Work
- Error Analysis: The authors suggest a need for a more detailed error analysis of word errors in the unadapted 10xRT pass compared to the 0.1xRT pass.
- Alternative Techniques: They propose extending their analysis to other topic identification techniques, such as Latent Semantic Analysis (LSA) and Latent Dirichlet Analysis (LDA).
Summary updated