Impact of word classing on shrinkage-based language models

Ruhi Sarikaya, Stanley F. Chen, Abhinav Sethy, Bhuvana RamabhadranInterspeech 2010 · DOI

Abstract

This study investigates the influence of various word-classing strategies on the performance of Model M, a shrinkage-based language model used in automatic speech recognition (ASR). While automatic clustering is effective for large-scale data, the integration of human-assigned labels—such as part-of-speech (POS) tags and domain-specific semantic categories—significantly reduces word error rates (WER). Our findings demonstrate that manual classing is particularly beneficial in data-sparse environments, yielding up to a 3% absolute improvement in WER.

Introduction

$N$-gram language models are fundamental to speech recognition but frequently suffer from data sparsity, where infrequent word sequences are poorly estimated. Class-based models mitigate this by grouping words into categories, allowing the model to predict class transitions rather than individual word transitions. Model M enhances this approach by using shrinkage (regularization) to interpolate between word-level and class-level distributions. Historically, Model M relied on automatically generated classes. This paper explores whether incorporating human knowledge through predefined semantic classes and POS tags can further optimize performance.

Methodology

The researchers evaluated several classing techniques within the Model M framework:

  1. Automatic Classing: Baseline grouping based on mutual information algorithms.
  2. Predefined Semantic Classes: Domain-specific categories (e.g., "CITY" for air travel).
  3. POS Classing: Standard grammatical tags.
  4. Hard vs. Soft Classing: Comparing models where words belong to a single class (hard) versus multiple classes (soft). Soft classing utilized a Viterbi approximation to determine the most likely class sequence.
  5. Hybrid "Merge" Strategies: Combining human-defined labels for domain-specific vocabulary with automatic clustering for the remaining words.

Experimental Setup

Two datasets were used to test the models under different conditions:

  • Air Travel Reservation: A large human-machine dialog dataset (~137,000 sentences) with 39 semantic classes.
  • Military/Medical Domain: A small, noisy human-human speech translation dataset (~5,400 sentences), representing a high-sparsity scenario.

Results and Discussion

The results indicate that human intuition is most valuable in the "long tail" of infrequent words where automatic algorithms lack sufficient data to form meaningful clusters.

Air Travel Domain

In the Air Travel task, the baseline trigram model achieved a WER of 19.9%. The introduction of hybrid strategies (Merge) and soft POS classing provided incremental improvements. The best performance was achieved using a hybrid approach, reducing the WER to 18.8%.

Table 1: Word Error Rate (WER) results for the Air Travel domain across various Model M configurations.

Military/Medical Domain

The impact of manual classing was most pronounced in the smaller Military/Medical dataset. While the baseline trigram WER was 15.7%, applying POS tags within Model M reduced the error rate to 12.7%. This 3% absolute reduction highlights the critical role of linguistic structure when training data is limited.

Table 2: Word Error Rate (WER) results for the Military/Medical domain, showing significant gains from POS classing.

Conclusion

This research demonstrates that while automatic word clustering is a powerful tool for large datasets, human-annotated semantic and syntactic information remains vital for robust language modeling, especially in data-constrained domains. Future work will focus on more efficient ways to combine these diverse classing strategies and reducing the computational overhead associated with soft classing.

Summary updated

BibTeX
@inproceedings{sarikaya2010impact,
  author = {Ruhi Sarikaya and Stanley F. Chen and Abhinav Sethy and Bhuvana Ramabhadran},
  title = {{Impact of word classing on shrinkage-based language models}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-515},
}
Figure