Searching the Audio Notebook: Keyword Search in Recorded Conversation

Peng Yu, Kaijiang Chen, Lie Lu, Frank Seide

Abstract

MIT's Audio Notebook added great value to the note-taking process by retaining audio recordings, e.g. during lectures or interviews. The key was to provide users ways to quickly and easily access portions of interest in a recording. Several non-speech-recognition based techniques were employed. In this paper we present a system to search directly the audio recordings by key phrases. We have identified the user requirements as accurate ranking of phrase matches, domain independence, and reasonable response time. We address these requirements by a hybrid word/phoneme search in lattices, and a supporting indexing scheme. We will introduce the ranking criterion, a unified hybrid posterior-lattice representation, and the indexing algorithm for hybrid lattices. We present results for five different recording sets, including meetings, telephone conversations, and interviews. Our results show an average search accuracy of 84%, which is dramatically better than a direct search in speech recognition transcripts (less than 40% search accuracy).

Introduction

Searching recorded conversations is significantly more difficult than searching broadcast news or dictated text. Conversational speech is spontaneous, overlapping, and often recorded in noisy environments, leading to Word Error Rates (WER) between 40% and 60%.

If a user searches a standard text transcript generated by an Automated Speech Recognition (ASR) system, they will likely miss more than half of the relevant hits because the "best guess" of the recognizer was wrong. Furthermore, users often search for proper names or technical jargon that are "Out-of-Vocabulary" (OOV)—meaning they aren't in the ASR system's dictionary at all. This research presents a system that is domain-independent, fast enough for large archives, and accurate despite high recognition error rates.

Methodology

The core innovation lies in how the audio is represented and indexed. Instead of searching a single text transcript, the system searches "lattices"—directed acyclic graphs representing many different hypotheses of what was said, along with their probabilities.

System architecture for hybrid word/phoneme indexing and search.

Key components of the system include:

  • Hybrid Word/Phoneme Approach: The system runs two recognizers in parallel: a Large Vocabulary Continuous Speech Recognition (LVCSR) system for common words and a phonetic recognizer for phonetic word fragments. This ensures that even if a word isn't in the dictionary, its sound (phonemes) can still be matched.
  • Unified Posterior Lattices: The authors convert both word and phoneme lattices into "posterior lattices." By including "node posteriors" (the probability that a word boundary exists at a specific time), they merge these two different data types into a single structure.
  • Two-Stage Indexing: To handle hundreds of hours of audio, the system uses a two-stage process. First, an M-gram based inverted index performs a fast "rough cut" to find promising audio segments. Second, a detailed "linear search" is performed on the lattices of those specific segments to rank the results.

Experimental Results

The system was evaluated on five diverse conversational corpora, including ICSI Meetings, Switchboard telephone speech, and in-house interviews. The results demonstrate that searching the raw ASR transcript is insufficient, yielding less than 40% accuracy. In contrast, the lattice-based hybrid approach provides a significant performance boost.

Search performance (FOM and THP) across different test sets and search modes.

As shown in the results table, the hybrid system achieved an average search accuracy (Figure of Merit) of 84.1% across all datasets. The phonetic track proved to be a powerful "safety net," successfully retrieving OOV words with roughly 73.8% accuracy—a task where standard transcript search fails completely. Furthermore, the two-stage indexing approach allowed the system to search nearly 200 hours of audio in under two seconds with negligible loss in accuracy.

Conclusion

By moving beyond simple text transcripts and utilizing a hybrid word/phoneme lattice approach, this system achieves high search accuracy even in the face of significant speech recognition errors. The integration of phonetic information effectively handles out-of-vocabulary terms, while the two-stage indexing scheme ensures the system remains scalable for large audio archives.

Summary updated

Abstract

MIT's Audio Notebook added great value to the note-taking process by retaining audio recordings, e.g. during lectures or interviews. The key was to provide users ways to quickly and easily access portions of interest in a recording. Several non-speech-recognition based techniques were employed. In this paper we present a system to search directly the audio recordings by key phrases. We have identified the user requirements as accurate ranking of phrase matches, domain independence, and reasonable response time. We address these requirements by a hybrid word/phoneme search in lattices, and a supporting indexing scheme. We will introduce the ranking criterion, a unified hybrid posterior-lattice representation, and the indexing algorithm for hybrid lattices. We present results for five different recording sets, including meetings, telephone conversations, and interviews. Our results show an average search accuracy of 84%, which is dramatically better than a direct search in speech recognition transcripts (less than 40% search accuracy).
BibTeX
@inproceedings{yu2005searching,
  author = {Peng Yu and Kaijiang Chen and Lie Lu and Frank Seide},
  title = {{Searching the Audio Notebook: Keyword Search in Recorded Conversation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=rJ-wSXbdbr},
}
Figure