Cache-Augmented Latent Topic Language Models for Speech Retrieval

Jonathan WintrodeNAACL 2015

TL;DR

This paper introduces a novel language model for speech retrieval that combines latent topic modeling with a cache-based mechanism to capture both broad subject matter and local word repetition. By jointly modeling these two aspects, the authors achieve improved speech retrieval performance over traditional N-gram models and standalone topic models across multiple languages.

What Problem Does This Paper Address?

The explosion of digital spoken media, particularly on platforms like YouTube, presents a significant challenge for information retrieval. While metadata like links and tags offer some access, searching within the spoken content remains difficult. Traditional speech retrieval systems rely on Large Vocabulary Continuous Speech Recognition (LVCSR) to transcribe audio into text, but LVCSR accuracy can be quite low in many languages, especially those with fewer resources. This errorful transcription severely hinders subsequent search. The paper aims to improve speech retrieval by leveraging the robustness of topic modeling, which is less sensitive to transcription errors, and by also capturing the tendency for words to repeat locally within spoken content.

Methods

The core contribution is a Cache-Augmented Latent Topic Model. This model extends the standard Latent Dirichlet Allocation (LDA) topic model by introducing a "cache" mechanism.

  • Latent Topic Model (Broad Context): Similar to LDA, the model assumes documents are generated from a mixture of latent topics. Each topic is represented by a distribution over words, capturing the general subject matter.
  • Cache Model (Local Context): The model also incorporates a cache that stores recently observed words within a document. This allows the model to capture local word repetition, a common phenomenon in spoken language.
  • Generative Process: For each word in a document, the model decides probabilistically whether to generate it from a latent topic or from the cache. This decision is governed by a document-specific cache probability, $\kappa(d)$.
  • Language Model Integration: The inferred topic distributions ($\Phi^{(t)}$) and cache probabilities ($\kappa^{(d)}$) are used to create document-specific language models. These models are then interpolated with a standard N-gram language model to improve speech recognition and retrieval performance. The interpolation weights are learned from the data.

Datasets

The experiments were conducted using corpora from the IARPA Babel Limited Language Pack (LP), No Target Audio Reuse (NTAR) condition. Specifically, the following languages were used:

  • Tagalog
  • Vietnamese
  • Zulu
  • Tamil

A 10-hour training set (audio and transcripts) was used for building acoustic and language models, as well as for estimating the topic model parameters. Held-out data was used for inference and evaluation.

Key Insights and Ideas

  • Complementary Nature of Topic and Repetition: The paper strongly suggests that both broad topic context and local word repetition are valuable, complementary sources of information for speech language modeling and retrieval.
  • Robustness to Transcription Errors: Topic models are inherently more robust to speech recognition errors than traditional N-gram models, making them suitable for low-resource languages or noisy audio.
  • Cache Captures Repetition: The cache mechanism effectively captures word repetition, and the assignment of cache states is highly correlated with word frequency, indicating that the model learns to use the cache for words that are likely to repeat.
  • Perplexity vs. Retrieval Performance: While incorporating the cache generally improves perplexity (a measure of how well a language model predicts a sample), the models with the lowest perplexity were not always the best for retrieval. This highlights the importance of evaluating models on the downstream task.

Key Results

  • Speech Retrieval Improvement: The cache-augmented topic language model improved overall speech retrieval (term detection) accuracy by 0.5 to 1.7% absolute over a typical N-gram language model across the tested languages.
  • Perplexity Boost: Interpolating with the cache provided a significant boost in perplexity for all languages and topic counts, demonstrating its effectiveness as a language model component.
  • Language Variation: The effectiveness of latent topics and the cache varied across languages, suggesting that language-specific characteristics influence the optimal model configuration. For instance, Vietnamese, with its syllable-level word units, showed higher cache usage.

Limitations and Future Work

  • Unigram Focus: The current model primarily adds unigram probabilities from topics and the cache to a 3-gram language model. The authors plan to extend the model to explicitly capture N-gram behavior within topics and the cache.
  • First-Pass Output Dependency: The current approach relies on the first-pass output of the LVCSR system. Keywords not present in this initial transcription cannot be recalled. Future work could involve using the model to re-decode the audio, potentially leading to larger gains.
  • Cache Size and Decay: Practical considerations for cache language models, such as optimal cache size and decay mechanisms for older words, were not deeply explored in this work but are noted as potential areas for future investigation.

Summary updated

BibTeX
@inproceedings{wintrode2015cache,
  author = {Jonathan Wintrode},
  title = {{Cache-Augmented Latent Topic Language Models for Speech Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure