Topic dependent language model based on topic voting on noun history

Welly Naptali, Masatoshi Tsuchiya, Seiichi NakagawaInterspeech 2009

TL;DR

This paper introduces a novel topic-dependent language model (TDC LM) that dynamically infers the topic of a conversation by "voting" on noun history. This approach significantly improves speech recognition performance by better capturing long-range dependencies compared to traditional n-gram models and even other topic-based methods like LDA, especially when interpolated with a standard language model.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems rely heavily on language models (LMs) to predict the next word. While traditional n-gram LMs excel at modeling short-term word dependencies, they struggle with long-range context, which is crucial for understanding the overall topic or theme of a discourse. This limitation can lead to errors, especially in longer utterances. Previous attempts to address this include cache models, trigger models, and topic mixture models. However, methods like Latent Dirichlet Allocation (LDA) for topic modeling often require pre-analysis of the test data, making them unsuitable for real-time ASR. This paper aims to develop a topic-dependent LM that can dynamically infer topics in real-time, thereby improving the accuracy of ASR systems by leveraging long-range contextual information.

Methods

The core innovation lies in how the TDC LM infers topics. First, Latent Semantic Analysis (LSA) is used to map nouns into a semantic vector space, revealing hidden relationships between them. This semantic space is then clustered using Vector Quantization (VQ) to define discrete topics. The "confidence" of a noun belonging to a specific topic is measured by its cosine similarity to the topic's centroid.

To determine the topic of a given word sequence, the TDC LM employs a "topic voting" mechanism. It observes a fixed-size window of preceding words (noun history) and votes for the topic based on the weighted occurrence of noun classes within that window. The confidence measure derived from LSA and VQ is used to weight these votes. This inferred topic is then integrated into the language model as an additional context, effectively creating a "topic-dependent class" (TDC) LM. The paper also details a backoff strategy for handling unseen events, similar to Katz backoff, but adapted to slide the topic window rather than just remove a word.

Datasets

The proposed method was evaluated on two large corpora:

  • Wall Street Journal (WSJ) corpus: This is a standard benchmark for ASR research. The paper used data from 1987-1989, with a vocabulary of approximately 20,000 words.
  • Mainichi Shimbun corpus: This is a Japanese news article corpus. The paper used data from 1991-1998 for training and January 1999 for testing. Due to the nature of Japanese text, a morphological analyzer (Mecab) was used to segment words. The vocabulary size was also around 20,000 words.

Key Insights and Ideas

  • Topic Voting for Real-time Inference: The paper's key contribution is a dynamic topic inference mechanism that doesn't require pre-analysis of the test set, making it suitable for real-time ASR.
  • Nouns as Topic Indicators: The approach leverages the fact that nouns are strong indicators of a discourse's topic, using LSA to capture their semantic relationships.
  • Confidence-Weighted Voting: Using a confidence measure for noun-topic assignments adds robustness to the voting process.
  • Integration into N-gram Framework: The inferred topic is seamlessly integrated as an additional context within an n-gram model, allowing for interpolation with traditional LMs.
  • Window Size Matters: The experiments consistently show that increasing the window size for topic voting generally improves performance, indicating the importance of longer-range context.

Key Results

The TDC LM demonstrated significant improvements in perplexity (a measure of how well a probability model predicts a sample) across both corpora when compared to various baseline models:

  • WSJ Corpus: The interpolated TDC LM achieved a perplexity of 96.0, outperforming the word-based 3-gram LM (111.6) by a relative improvement of 13.98%. It also outperformed cache-based and LDA-based topic models.
  • Mainichi Shimbun Corpus: The interpolated TDC LM achieved a perplexity of 55.9, a relative improvement of 15.56% over the word-based 3-gram LM (66.2).

Interestingly, the "stand-alone" TDC LM (without interpolation) performed worse than traditional n-gram models on the WSJ corpus. The authors attribute this to the corpus being shrunk for each topic during training. However, when interpolated with a word-based 3-gram LM, the TDC model consistently showed superior performance.

Limitations and Future Work

  • Stand-alone Performance: The stand-alone TDC LM's performance degradation on the WSJ corpus suggests that the topic-specific training data might be too sparse.
  • LSA Dimension Size: The paper used a fixed LSA dimension size (200) and suggests further analysis of its impact.
  • VQ Clustering: The use of VQ for topic definition could potentially be improved by using alternative methods like Probabilistic LSA (PLSA) or LDA.
  • Soft-Decision Voting: The current voting mechanism is hard-decision; exploring soft-decision voting could be beneficial.
  • ASR System Experiments: The authors express a desire to conduct actual ASR system experiments to validate the real-world impact of their model.

Summary updated

BibTeX
@inproceedings{naptali2009topic,
  author = {Welly Naptali and Masatoshi Tsuchiya and Seiichi Nakagawa},
  title = {{Topic dependent language model based on topic voting on noun history}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure