TL;DR
This paper explores how to improve spoken document retrieval (SDR) by combining different types of evidence derived from speech. The authors propose and investigate methods for integrating automatically generated topic labels and vocabulary-independent spoken term detection (STD) with standard Automatic Speech Recognition (ASR) outputs, aiming to make SDR more robust across various domain-matching conditions.
What Problem Does This Paper Address?
Spoken Document Retrieval (SDR) aims to find information within audio or video recordings. Traditionally, this relied on expensive human-created metadata like topic labels or manual transcripts. The advent of Automatic Speech Recognition (ASR) made it possible to generate transcripts automatically, significantly lowering the cost. However, ASR systems often struggle with domain-specific vocabulary, leading to poor transcript quality and thus impacting retrieval performance. This paper addresses the challenge of improving SDR when ASR transcripts are imperfect, by exploring how to leverage additional, potentially more robust, evidence sources.
The core problem is that ASR systems have a fixed vocabulary, and if the spoken content uses words outside this vocabulary, the transcripts will be inaccurate. This "domain mismatch" is a significant hurdle for SDR. The paper investigates how to combine evidence from two sources that can help overcome this:
- Automatic Topic Classification: Using text classification on ASR transcripts to assign topic labels, which can act as additional indexable information.
- Vocabulary-Independent Spoken Term Detection (STD): Systems that can search for query words even if they are not in the ASR's vocabulary, by working with phonetic representations.
The paper aims to understand how combining these sources can improve retrieval, especially in scenarios where the ASR system's vocabulary is poorly matched to the domain of the spoken content.
Methods
The paper proposes and investigates several methods for combining evidence for SDR:
- Creating Additional Indexable Text: This is the simplest approach, where automatically generated topic labels (keywords) are treated as an additional field for indexing documents. This method was used in previous work and showed a relative improvement in mean average precision.
- Runtime Query Classification for SDR: Instead of classifying documents, this method classifies the user's query at search time. The resulting label assignments are then used to rank documents. The intuition is that users might use language more similar to the interviewees than to the thesaurus builders used for classification.
- Class Guided Document Expansion: This approach uses classification output as "seed text" for document expansion. The idea is that class labels might represent a more semantically dense representation of a document's topical content than the ASR transcript alone, potentially leading to better document-centered term expansion.
- Unconstrained Term Detection for SDR: This section discusses the challenge of combining STD with topical relevance IR systems. A proposed solution involves normalizing the STD system's score for out-of-vocabulary (OOV) query terms by a function of the STD system's score on in-vocabulary terms. This normalization aims to leverage the ASR transcript as a "ground truth" for in-vocabulary terms to better estimate confidence for OOV terms.
- Classifying with Temporal Evidence: This work focuses on improving the production of evidence. It explores using temporal information within spoken document collections.
- Time-shifted classification (TSC): Labels are assigned to subsequent segments based on the current segment's content, capturing the idea that temporal context can be predictive.
- Temporal label weighting (TLW): Non-local temporal information (e.g., the chronological order of an interview) is used to bias label assignments.
- Classifying across languages: This method addresses domain mismatch in multilingual collections by using training data from one language to classify documents in another, employing cross-language text classification techniques.
- Vocabulary Independent Spoken Utterance Retrieval: This research explores a low-resource approach to utterance retrieval using phonetic lattices and expected posterior counts of n-grams as indexing units.
Datasets
The primary dataset used in this research is the MALACH collection. This collection consists of approximately 1,000 hours of conversational speech from 400 interviews with Holocaust survivors. The interviews are segmented into 8,104 documents, each manually assigned topic labels from a thesaurus of roughly 40,000 descriptors. The MALACH collection has been used in previous cross-language speech retrieval evaluations (CLEF CLSR track). The paper notes that a larger subset of the MALACH collection (roughly 116,000 hours) is also available and provides training examples for classification.
Key Insights and Ideas
- Domain Mismatch is Key: The paper strongly emphasizes that the utility of different evidence sources (like topic labels and STD) is highly dependent on how well the ASR system's vocabulary matches the domain of the spoken content.
- Complementary Strengths: Topic classification offers strong retrieval improvements when the domain is well-matched, but struggles with domain mismatch. Conversely, STD is most promising precisely when the ASR vocabulary is poorly matched. Combining them could offer a more robust solution across different scenarios.
- Temporal Context Matters: For collections like oral histories, the temporal ordering of segments within an interview provides valuable information that can significantly improve classification accuracy.
- Phonetic Uncertainty is Informative: For spoken term detection, explicitly modeling the uncertainty in ASR predictions (via phonetic lattices) can lead to better performance than relying solely on the best transcription.
- Cross-Language Transfer is Possible: Even without direct training data for a specific language, metadata can be produced by leveraging existing data in another language through cross-language text classification.
Key Results
- Indexable Text Improvement: Using automatically produced topic labels as indexable text yielded a relative improvement in mean average precision of 40.6% over a baseline without keywords in previous work.
- Temporal Classification Gains: A combined Time-Shifted Classification (TSC) and Temporal Label Weighting (TLW) approach on the MALACH data showed relative improvements in mean average precision of 8.0% for conceptual thesaurus terms and 14.2% for geographic thesaurus terms.
- Cross-Language Classification Performance: A cross-language text classification approach achieved 73% of the monolingual English baseline performance on conceptual topic assignment.
- Vocabulary Independent Retrieval: An approach for vocabulary independent spoken utterance retrieval significantly improved mean average precision over a 1-best phone sequence baseline on five human languages.
Limitations and Future Work
- Domain Mismatch Handling: While the paper proposes combining evidence to address domain mismatch, the exact optimal combination strategies and their performance across a wide spectrum of domain-matching conditions are still areas for investigation.
- STD and Topical Relevance Integration: The paper acknowledges that it is "not yet clear how best to combine a STD and topical relevance IR system." The proposed normalization method for STD scores is an initial step, but further research is needed.
- Phonetic Query Degradation/Expansion: The proposed methods for improving spoken term detection (phonetic query degradation and expansion) are still under investigation. The paper notes potential disadvantages like requiring examples or assuming the degradation process is well-represented by a few examples.
- Production of Evidence: The paper highlights work on improving the production of evidence (e.g., temporal classification, cross-language classification, phonetic STD), suggesting that further improvements in these underlying technologies will benefit the overall SDR system.
- New Resources: The paper mentions that the MALACH collection, while valuable, is only a subset of a larger dataset, implying that future work could leverage the full dataset for more comprehensive training and evaluation.
Summary updated