Spoken document retrieval from call-center conversations

Jonathan Mamou, David Carmel, Ron HoorySIGIR 2006 · DOI

TL;DR

This paper tackles the challenge of retrieving information from noisy call-center conversations, where standard speech recognition systems struggle. They propose using "word confusion networks" (WCNs) – a richer output from speech recognition that includes multiple word hypotheses and their probabilities – to significantly improve retrieval accuracy, even with very high error rates.

What Problem Does This Paper Address?

Call centers are crucial for customer service, but the audio data they generate is notoriously difficult for automatic speech recognition (ASR) systems. Factors like low recording quality, background noise, and spontaneous speech patterns lead to high word error rates (WERs), often exceeding 30% and sometimes reaching 50% or more. This makes traditional text-based information retrieval (IR) on ASR transcripts highly unreliable, as errors in transcription directly translate to missed information and inaccurate search results. The paper aims to bridge this gap by developing a retrieval system that can effectively handle these noisy transcripts.

Methods

The core of the paper's contribution lies in leveraging the output of ASR systems beyond the single "best guess" transcript. Instead of just using the 1-best path from the ASR, they utilize Word Confusion Networks (WCNs). A WCN represents a word lattice in a more compact form and, crucially, provides not only word hypotheses but also their posterior probabilities (how likely a word is given the audio) and their rank among other hypotheses at a given time offset.

The authors propose two main enhancements to standard IR techniques:

  1. WCN Indexing: Instead of indexing only the 1-best transcript, they index all hypotheses within the WCN. This expands the potential search space but also introduces more noise.
  2. WCN-based Ranking: They develop a novel ranking model that goes beyond simple term frequency. This model incorporates:
  3. Confidence Levels: Terms with higher posterior probabilities are given more weight.
  4. Term Rank: Terms that are ranked higher among hypotheses at a given offset are also boosted.
  5. Lexical Affinities: They also consider the proximity of related query terms.

This allows the system to "trust" terms that the ASR is more confident about and to prioritize terms that appear earlier in the hypothesis list.

Datasets

The experiments were conducted on a corpus of 2236 call-center conversations from IBM's internal customer support service during 2001. This data included manual transcripts and speaker segmentation. The corpus was split into training and testing sets for the ASR system. The authors also used 120 queries designed to represent typical call-center topics for evaluation.

Key Insights and Ideas

  • Beyond the 1-Best: The most significant insight is that simply using the 1-best ASR transcript is a bottleneck. The additional information in WCNs (probabilities and ranks) is crucial for improving retrieval.
  • Noise Tolerance: The paper demonstrates that even with very high word error rates (up to 50-60%), a well-designed retrieval system using WCNs can still achieve reasonable performance, particularly in terms of recall.
  • Confidence and Rank Matter: The proposed ranking model, which combines posterior probabilities and hypothesis ranks, is shown to be more effective than simpler approaches that ignore these factors.
  • Recall vs. Precision Trade-off: Indexing all WCN hypotheses significantly improves recall (finding more relevant documents) but can decrease precision (returning more irrelevant documents). The ranking model helps to mitigate this precision drop.

Key Results

  • MAP Improvement: Using WCNs with the proposed ranking model (all WCN CL boost) consistently outperformed the 1-best transcript baseline across various WER levels. For example, at a WER of 40.5%, the MAP for 1-best WCN was 0.76, while the "all WCN CL boost" model achieved higher MAP (though the exact number isn't directly comparable in the provided figures, the trend is clear).
  • Recall Boost: Indexing all WCN hypotheses (all WCN) significantly boosted recall compared to the 1-best WCN approach, especially at higher WERs.
  • Robustness to High WER: Even at a WER of around 50-60%, the MAP remained at a reasonable level, indicating the system's robustness. For instance, at WER ~50%, precision was still around 0.5, and recall was around 0.8.
  • Index Size: While indexing all WCN hypotheses increased the index size by about 20 times compared to the 1-best path, it was only about three times larger than the 1-best path index.

Limitations and Future Work

  • Out-of-Vocabulary (OOV) Words: The current system assumes all query terms are within the ASR's vocabulary. Handling OOV queries, where ASR might replace words with probable alternatives, is noted as a limitation and a direction for future research.
  • Precision Drop with Full Indexing: While indexing all hypotheses improves recall, it leads to a decrease in precision. The ranking model helps, but further work might be needed to optimize this trade-off.
  • Real-world Application: The authors suggest that real-world applications might require special treatment for OOV queries.

Summary updated

BibTeX
@inproceedings{mamou2006spoken,
  author = {Jonathan Mamou and David Carmel and Ron Hoory},
  title = {{Spoken document retrieval from call-center conversations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2006},
  doi = {10.1145/1148170.1148183},
}
Figure