Integration of cache-based model and topic dependent class model with soft clustering and soft voting

Welly Naptali, Masatoshi Tsuchiya, Seiichi NakagawaInterspeech 2010

TL;DR

This paper introduces a novel language modeling approach that combines topic-based modeling with cache-based modeling to improve speech recognition accuracy. By integrating these techniques with soft clustering and soft voting, the authors demonstrate significant reductions in perplexity, a key metric for language model performance, on both English and Japanese corpora.

What Problem Does This Paper Address?

Traditional n-gram language models (LMs) in speech recognition primarily rely on word sequence history. However, human understanding of language also heavily involves grasping the underlying topic of a conversation. Existing topic-dependent class (TDC) models attempt to capture this topical information by analyzing noun relationships, but they face challenges: they struggle with modeling common functional words and their performance degrades as the number of topics increases, often requiring interpolation with a general n-gram model. This paper aims to address these limitations by enhancing the TDC model.

Methods

The core contribution is the integration of two key enhancements to the Topic Dependent Class (TDC) language model:

  • Cache-based LM Integration: The authors incorporate a cache-based LM, which models the tendency for words to reappear within a recent history, into the TDC framework. This is achieved through "unigram scaling," a method that scales the unigram probability of the TDC model using the cache-based model. This addresses the TDC's weakness in modeling recurring words.
  • Soft Clustering and Soft Voting: To overcome the issue of performance degradation with more topics and to make topic assignment more robust, the authors introduce "soft clustering" and "soft voting."
    • Soft Clustering: Instead of assigning a noun to a single topic (hard clustering), soft clustering allows a noun to belong to multiple topics with varying degrees of confidence. This is quantified by a confidence measure derived from cosine similarity.
  • Soft Voting: This technique allows the model to consider multiple "best" topics (k-best) rather than just the single most probable one. Applying soft voting during training effectively increases the training corpus size by allowing word sequences to contribute to multiple topics. During testing, it transforms the TDC model into a mixture model, averaging probabilities across the selected topics.

The paper also explores two ways of combining the TDC, cache-based, and n-gram models: TDCCACHE+NGRAM (scaling TDC before interpolation) and TDC+NGRAMCACHE (scaling after interpolation).

Datasets

The experiments were conducted on two standard corpora:

  • Wall Street Journal (WSJ) Corpus: English news text, split into training (36,754,891 words) and test (336,096 words) sets.
  • Mainichi Shimbun Corpus: Japanese newspaper text, with a larger training set (207,215,663 words) and a smaller test set (385,863 words).

The paper notes the use of ARPA's official "200.nvp" vocabulary for WSJ and the Mecab toolkit for Japanese text processing.

Key Insights and Ideas

  • Complementary Strengths: The cache-based LM and TDC model capture different aspects of language – recurring words versus topical words – and their combination yields significant improvements, supporting the idea that they are complementary.
  • Robustness through Softness: Moving from hard assignments (single topic, single best prediction) to soft assignments (multiple topics, multiple best predictions) significantly improves model performance, particularly for the stand-alone TDC model. This suggests that linguistic phenomena are often nuanced and not strictly categorical.
  • Self-Sufficiency of TDC: The soft clustering and soft voting techniques enable the TDC model to perform well on its own, without needing to be interpolated with a general word-based n-gram model, thus solving a key limitation of previous TDC approaches.

Key Results

  • WSJ Corpus:
    • The TDC+NGRAMCACHE model achieved a perplexity of 84.9, representing a 23.9% relative improvement over the word-based 3-gram baseline (111.6) and an 11.8% improvement* over the previous TDC model (96.2).
  • With soft clustering and soft voting, the stand-alone TDC model achieved a perplexity of 93.2 (compared to 175.8 for hard voting), a 47.0% relative improvement. The interpolated model also improved from 95.8 to 90.8.
  • Mainichi Shimbun Corpus:
    • The TDCCACHE+NGRAM model achieved a perplexity of 53.3, a 25.7% relative improvement* over the word-based 3-gram baseline (71.7).
  • The TDC+NGRAMCACHE model achieved 58.0, a 19.1% improvement* over the word-based 3-gram baseline.

Limitations and Future Work

  • Fixed k: The current implementation uses a fixed number of "k-best" topics for soft clustering and voting. The authors suggest that this might lead to the inclusion of unimportant word sequences and increased memory requirements.
  • Threshold-based Approach: As a future direction, they propose using a threshold based on confidence measures instead of a fixed k, which could be more efficient.
  • ASR Evaluation: The authors express a desire to evaluate the proposed TDC LM on an Automatic Speech Recognition (ASR) system to assess its real-world impact.

Summary updated

BibTeX
@inproceedings{naptali2010integration,
  author = {Welly Naptali and Masatoshi Tsuchiya and Seiichi Nakagawa},
  title = {{Integration of cache-based model and topic dependent class model with soft clustering and soft voting}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure