Abstract
Building domain-specific language models often suffers from a lack of specialized data. While the web offers a massive corpus, standard filtering methods often over-select "easy," high-probability sentences that skew the model's distribution. This paper introduces an incremental selection algorithm based on relative entropy that picks a subset of sentences to match the target domain's distribution, resulting in smaller, more accurate models using only 10% of the available raw data.
Introduction
When training language models (LMs) for specific tasks—such as medical transcriptions or technical support—researchers face a significant "data sparsity" problem. There simply isn't enough in-domain text to train robust models. A common solution is to harvest text from the World Wide Web; however, web data is noisy and linguistically diverse, often differing significantly from the target domain in style and content.
Existing "rank-and-select" methods typically score web sentences based on how well they fit an existing in-domain model (e.g., using perplexity). However, this creates a "rich get richer" effect: the algorithm keeps picking sentences that are already highly probable in the training set (like "Yes," or "Okay"). This skews the final model's distribution, making it perform poorly on the diverse, lower-probability sentences that are actually crucial for robust speech recognition.
Methodology
The authors propose an Incremental Selection (ItSel) algorithm. Instead of looking at sentences in isolation, they evaluate how adding a sentence changes the entire distribution of the selected set.
- Relative Entropy (KL Divergence) Criterion: The goal is to select a subset of sentences such that the relative entropy between the subset's distribution and the target in-domain distribution is minimized.
- Greedy Selection: The algorithm scans the large web corpus and adds a sentence only if it decreases the relative entropy of the current pool.
- Fast Computation: Calculating relative entropy for every sentence in a billion-word corpus is computationally expensive. The authors derived a simplified formula that allows for rapid updates, making the process feasible for massive datasets.
- Resequencing: To avoid the pitfalls of a purely greedy approach (where the order of sentences matters), they randomize the corpus and perform multiple passes, including a "reverse scan" to prune sentences that no longer contribute to the distributional match.
Key Insights
- Distributional Similarity: Matching the "shape" of the language is more important than just picking "good" sentences. Standard perplexity filtering (PPLSel) focuses on the center of the distribution, while the proposed method (ItSel) maintains the necessary tail of the distribution.
- The "Imbalance" Problem: As shown in Figure 1, standard filtering methods (PPLSel) lead to a high "relative entropy imbalance," meaning they over-represent certain parts of the language and under-represent others. The proposed method keeps this imbalance near zero.
- Efficiency and Model Size: You don't need a billion words to build a great model. By selecting the right 10% of data, you can outperform a model trained on the entire, noisy 100%. The resulting models were much leaner; for the 850M word case, the number of bigrams and trigrams was reduced by a factor of 7 to 9 compared to using the full web data.
Experimental Results
The proposed method consistently outperformed both the "No Web" baseline and standard filtering techniques across several metrics:
- Perplexity (PPL): On the 850M word corpus, the proposed method achieved a perplexity of 37.1, compared to 49.6 for the baseline and 38.2 for the "All Web" approach.
- Word Error Rate (WER): In ASR testing, the method reduced the WER significantly. As shown in the results table below, the proposed method yields the lowest error rates across different training set sizes (10K, 20K, and 40K sentences).
- Data Selection Efficiency: The algorithm achieved these results using only about 9-12% of the available web data, demonstrating that intelligent selection is far superior to brute-force data inclusion.
Conclusion
The relative entropy-based incremental selection method provides a robust way to harvest domain-specific text from large, noisy corpora. By focusing on distributional matching rather than individual sentence scoring, it produces language models that are both more accurate and significantly more efficient in terms of memory and computation.
Summary updated