Introduction
Spoken content retrieval (SCR) serves as the search engine for audio data, enabling users to locate specific information within vast spoken archives. Traditional systems transcribe audio into lattices—graphical representations of possible word sequences—and search these for query terms. However, two primary challenges persist: the Out-of-Vocabulary (OOV) problem, where queries contain terms not present in the system's lexicon, and recognition errors, where the acoustic signal is misidentified during transcription. While previous research addressed these issues for single-word, in-vocabulary queries, this paper proposes a robust framework for open-vocabulary retrieval capable of handling multi-word queries of varying lengths.
Proposed Methodology
The authors introduce a multi-stage pipeline designed to enhance retrieval accuracy by leveraging acoustic feature similarity alongside traditional lattice-based searching.
- First-Pass Retrieval: The system initially searches word or subword (syllable/phone) lattices to generate a ranked list of potentially relevant utterances based on a relevance score.
- Acoustic Feature Similarity: For query terms, the system extracts underlying acoustic features (MFCCs) from audio segments. It employs Dynamic Time Warping (DTW) to measure how "sound-alike" different utterances are, regardless of their textual transcription.
- Pseudo-Relevance Feedback (PRF): Assuming the top-ranked results from the first pass are likely correct, the system boosts the scores of other utterances in the list that exhibit high acoustic similarity to these "pseudo-correct" examples.
- Graph-based Re-ranking: A graph is constructed where each retrieved utterance is a node, and edges represent acoustic similarity. A Random Walk algorithm propagates relevance scores across the graph. If an utterance is acoustically similar to many other highly-ranked utterances, its score is increased.
Unlike prior work limited to single words, this method formulates similarity across n-grams, allowing it to handle complex, multi-word queries.
Experimental Setup
The framework was evaluated using the NTU Lecture Corpus, which contains 45 hours of spontaneous Mandarin Chinese lectures. This dataset is particularly challenging as it includes a mix of Mandarin and English terms. The testing set comprised 33 hours of audio, evaluated against 275 in-vocabulary (IV) Chinese queries and 110 OOV English queries.
Key Insights
- Subword Units for OOV: While word-based lattices are highly accurate for known terms, subword units like syllables and phones are essential for retrieving OOV content.
- Acoustic Consistency: Acoustic features remain stable even when transcription fails. PRF and graph-based re-ranking leverage this consistency to "rescue" relevant results that were initially ranked low due to recognition errors.
- Complementary Context: PRF provides local context by comparing utterances to the top-tier results, while the graph-based approach considers the global structure of the entire retrieved list.
Results and Performance
Performance was measured using Mean Average Precision (MAP). The results demonstrate that the PRF+Graph re-ranking consistently outperforms the first-pass retrieval across various acoustic models, including Speaker Independent (SI), Speaker Adaptive (SA), and Speaker Dependent (SD).
For Speaker Independent (SI) models, the first-pass MAP of 56.18% for word-based retrieval improved to 65.54% with the integrated PRF+Graph approach. Similarly, for OOV queries using syllable-based lattices, the full pipeline pushed the MAP from 42.40% to 53.56%. The re-ranking approach proved robust, showing gains in every experimental configuration.
Conclusion
The proposed framework significantly improves open-vocabulary spoken content retrieval by integrating acoustic similarity into a multi-stage re-ranking process. While effective, the authors note that re-ranking is limited by the recall of the first-pass retrieval. Future work will focus on improving initial search recall and exploring the computational efficiency of these methods for real-time applications.
Summary updated