Enhanced word classing for model M

Stanley F. Chen, Stephen M. ChuInterspeech 2010

TL;DR

This paper introduces a novel word classing method tailored for the "Model M" language model, significantly improving speech recognition accuracy. By optimizing a modified objective function that better reflects Model M's structure and incorporating word n-gram features, the new approach achieves up to a 3.0% absolute reduction in word error rate compared to traditional methods.

What Problem Does This Paper Address?

Language models, particularly n-gram models, are crucial for speech recognition. "Model M" is a powerful class-based n-gram model that has shown great promise. However, the standard method for creating word classes (bigram mutual information clustering) is based on a simpler language model, creating a mismatch. This discrepancy means that the word classes generated might not be optimal for Model M, limiting its full potential. The paper aims to bridge this gap by developing a word classing technique specifically designed to enhance Model M's performance.

Methods

The core of the paper lies in modifying the objective function used for word classing. Instead of optimizing for a simple bigram model, the authors propose optimizing for a version of Model M itself, or a simplified approximation. Key methodological contributions include:

  • Modified Objective Function: The paper introduces a "joint modeling" approach, which is a more direct reflection of how Model M uses word and class information. This involves adapting the objective function to account for word n-gram features, which are central to Model M's design but absent in the traditional IBM classing method.
  • Handling Unseen Data: To ensure the classing method generalizes well to unseen data (test sets), the authors incorporate a "leaving-one-out" estimation technique. This involves estimating the likelihood of unseen word n-grams by considering different scenarios: when the n-gram is seen, when its class n-gram is seen, or when neither is seen.
  • Improved Search Algorithm: The paper also discusses enhancements to the search algorithm for finding optimal word classes. While the standard exchange algorithm is used, the authors suggest incorporating class merge and split moves to escape local optima more effectively.

Datasets

The experiments were conducted using the Wall Street Journal (WSJ) dataset.

  • Training Data: Sentences were randomly selected from the 1993 CSR-III Text corpus, with training sets of varying sizes: 1k, 10k, 100k, and 900k sentences (approximately 23 million words for the largest set).
  • Development Set: A matched development set of 2.5k sentences (64k words) was used for selecting classing algorithm parameters.
  • Test Vocabulary: The vocabulary comprised the union of training vocabularies and a 20k word "closed" test vocabulary from the first WSJ CSR corpus, totaling about 21k words.
  • Speech Recognition Evaluation: Acoustic development and evaluation sets were also used, derived from Broadcast News data.

Key Insights and Ideas

  • Mismatch Matters: The paper strongly suggests that the mismatch between the classing objective function and the target language model (Model M) is a significant bottleneck. Tailoring the classing method to the target model yields substantial improvements.
  • Word N-grams are Key: Incorporating word n-gram features directly into the classing objective function is crucial for Model M's performance, as Model M itself relies heavily on these features.
  • Joint Modeling for Better Generalization: The shift to a joint modeling objective function, rather than purely conditional, helps in estimating performance on unseen data more effectively.
  • Search Algorithm Enhancements: While the objective function is primary, improvements in the search strategy (like merge/split moves) can further refine class discovery.

Key Results

  • Significant Word Error Rate Reduction: The proposed enhanced word classing method achieved up to a 1.1% absolute improvement over the baseline IBM classing method on Model M.
  • Overall Gains: Compared to a standard Katz-smoothed trigram model, the new approach resulted in a total gain of up to 3.0% absolute in word error rate, which the authors claim is the largest such gain reported for a class-based language model.
  • Perplexity Improvements: Table 1 shows that the new algorithm consistently achieves lower log perplexity on the matched development set across various training set sizes and modifications, indicating better language modeling.
  • Impact of Modifications: Table 1 also demonstrates that changes to the objective function generally have a larger impact on performance than changes to the search algorithm.

Limitations and Future Work

  • Scalability to Very Large Datasets: The authors acknowledge that it remains to be seen whether the observed gains will scale to training sets of hundreds of millions of words and larger.
  • Coefficient Tuning: The paper mentions using a coefficient ($\beta$) to tune the impact of the "Cunseen" term, which controls the number of word classes. This suggests that finding the optimal number of classes might still require careful tuning.
  • Future Directions: The paper implies that further exploration of class merge and split moves, as well as investigating the impact of different discount estimation methods, could be beneficial. The potential application of these improved word classes to other NLP tasks beyond speech recognition is also highlighted.

Summary updated

BibTeX
@inproceedings{chen2010enhanced,
  author = {Stanley F. Chen and Stephen M. Chu},
  title = {{Enhanced word classing for model M}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure