Text data acquisition for domain-specific language models

Abhinav Sethy, Panayiotis G. Georgiou, Shrikanth NarayananEMNLP 2006 · OpenReview

Abstract

Building domain-specific language models often suffers from a lack of specialized data. While the web offers a massive corpus, standard filtering methods often over-select "easy," high-probability sentences that skew the model's distribution. This paper introduces an incremental selection algorithm based on relative entropy that picks a subset of sentences to match the target domain's distribution, resulting in smaller, more accurate models using only 10% of the available raw data.

Introduction

When training language models (LMs) for specific tasks—such as medical transcriptions or technical support—researchers face a significant "data sparsity" problem. There simply isn't enough in-domain text to train robust models. A common solution is to harvest text from the World Wide Web; however, web data is noisy and linguistically diverse, often differing significantly from the target domain in style and content.

Existing "rank-and-select" methods typically score web sentences based on how well they fit an existing in-domain model (e.g., using perplexity). However, this creates a "rich get richer" effect: the algorithm keeps picking sentences that are already highly probable in the training set (like "Yes," or "Okay"). This skews the final model's distribution, making it perform poorly on the diverse, lower-probability sentences that are actually crucial for robust speech recognition.

Methodology

The authors propose an Incremental Selection (ItSel) algorithm. Instead of looking at sentences in isolation, they evaluate how adding a sentence changes the entire distribution of the selected set.

  1. Relative Entropy (KL Divergence) Criterion: The goal is to select a subset of sentences such that the relative entropy between the subset's distribution and the target in-domain distribution is minimized.
  2. Greedy Selection: The algorithm scans the large web corpus and adds a sentence only if it decreases the relative entropy of the current pool.
  3. Fast Computation: Calculating relative entropy for every sentence in a billion-word corpus is computationally expensive. The authors derived a simplified formula that allows for rapid updates, making the process feasible for massive datasets.
  4. Resequencing: To avoid the pitfalls of a purely greedy approach (where the order of sentences matters), they randomize the corpus and perform multiple passes, including a "reverse scan" to prune sentences that no longer contribute to the distributional match.

Key Insights

  • Distributional Similarity: Matching the "shape" of the language is more important than just picking "good" sentences. Standard perplexity filtering (PPLSel) focuses on the center of the distribution, while the proposed method (ItSel) maintains the necessary tail of the distribution.
  • The "Imbalance" Problem: As shown in Figure 1, standard filtering methods (PPLSel) lead to a high "relative entropy imbalance," meaning they over-represent certain parts of the language and under-represent others. The proposed method keeps this imbalance near zero.
Figure 1: Relative entropy imbalance vs. number of sentences. The proposed method (ItSel) maintains a much more balanced distribution than perplexity-based selection (PplSel).
  • Efficiency and Model Size: You don't need a billion words to build a great model. By selecting the right 10% of data, you can outperform a model trained on the entire, noisy 100%. The resulting models were much leaner; for the 850M word case, the number of bigrams and trigrams was reduced by a factor of 7 to 9 compared to using the full web data.

Experimental Results

The proposed method consistently outperformed both the "No Web" baseline and standard filtering techniques across several metrics:

  • Perplexity (PPL): On the 850M word corpus, the proposed method achieved a perplexity of 37.1, compared to 49.6 for the baseline and 38.2 for the "All Web" approach.
  • Word Error Rate (WER): In ASR testing, the method reduced the WER significantly. As shown in the results table below, the proposed method yields the lowest error rates across different training set sizes (10K, 20K, and 40K sentences).
Figure 5: Word Error Rate (WER) results showing the proposed method consistently yields the lowest error across different training set sizes.
  • Data Selection Efficiency: The algorithm achieved these results using only about 9-12% of the available web data, demonstrating that intelligent selection is far superior to brute-force data inclusion.

Conclusion

The relative entropy-based incremental selection method provides a robust way to harvest domain-specific text from large, noisy corpora. By focusing on distributional matching rather than individual sentence scoring, it produces language models that are both more accurate and significantly more efficient in terms of memory and computation.

Summary updated

Abstract

The language modeling community is showing a growing interest in using large collections of text mined from the World Wide Web (WWW) to supplement sparse in-domain text resources. However, in most cases the style and content of the text harvested from these corpora differs significantly from the specific nature of these domains. In this paper we present a relative entropy (r.e.) based method to select relevant subsets of sentences whose distribution in an n-gram sense matches the domain of interest. Using simulations, we provide an analysis of how the proposed scheme outperforms filtering techniques proposed in recent language modeling literature on mining text from the web. A comparative study is presented using a text collection of over 800M words collected from the WWW. Experimental results show that by using the proposed subset selection scheme we can get performance improvement in both Word Error Rate (WER) and Perplexity (PPL) over the models built from the entire collection by using just 10% of the data. Improvements in data selection also translated to a significant reduction in the vocabulary size as well as the number of estimated parameters in the adapted language model.
BibTeX
@inproceedings{sethy2006text,
  author = {Abhinav Sethy and Panayiotis G. Georgiou and Shrikanth Narayanan},
  title = {{Text data acquisition for domain-specific language models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=r1VBJGMOZr},
}
Figure