Abstract
Statistical language models often struggle with domain adaptation when target-domain data is scarce. This paper presents the Doubly Hierarchical Pitman-Yor Process Language Model (DHPYLM), a Bayesian framework that integrates a latent general model with domain-specific models. By leveraging a Graphical Pitman-Yor Process, the model enables principled sharing of statistical strength across domains, consistently outperforming standard baselines and demonstrating significant data efficiency gains.
Introduction
In statistical language modeling, domain adaptation remains a significant challenge. Building specialized systems—such as speech recognizers for medical or legal transcripts—often requires training on small, domain-specific datasets. While large general-purpose corpora (e.g., web text) are readily available, models trained on them often fail to capture specialized vocabulary. Conversely, models trained solely on small specific datasets tend to overfit. The DHPYLM addresses this by providing a unified Bayesian framework to "smooth" specific models using general data without losing domain-specific nuances.
Methodology
The DHPYLM builds upon the Hierarchical Pitman-Yor Process Language Model (HPYLM), a state-of-the-art Bayesian approach to Kneser-Ney smoothing. The authors introduce a second layer of hierarchy to create a "doubly hierarchical" structure.
The Latent Model and Domain Hierarchy
The framework posits a latent (unobserved) general language model that represents the underlying structure of the language. Each specific domain (e.g., news, transcripts) is represented by its own HPYLM. These domain models are coupled through the shared latent model, which acts as a parent distribution.
Recursive Back-off and Inference
A key innovation is the recursive back-off mechanism. When the model encounters a low-frequency word in a specific context, it can back off in two directions: to a shorter context within the same domain (e.g., trigram to bigram) or to the full context in the latent general model. This is formalized through the Graphical Pitman-Yor Process (GPYP), where the base distribution of one process is a mixture of others. Inference is performed using a complex Gibbs sampling algorithm metaphorically described as the Multi-Floor Chinese Restaurant Franchise (MFCRF).
Experimental Setup
The model was evaluated using three primary datasets:
- Brown Corpus: ~1 million words of general written English (out-of-domain).
- SOU Corpus: ~370k words of US State of the Union addresses (in-domain).
- AMI Meeting Corpus: ~800k words of transcribed meeting conversations (in-domain).
Results and Discussion
The researchers utilized perplexity—a measure of how well a probability model predicts a sample—as the primary metric. The DHPYLM consistently outperformed both baseline models trained only on domain data and "union" models trained on a simple mixture of all data.
Data Efficiency and Statistical Strength
A critical insight from the results is the model's data efficiency. As shown in the contour plots, adding out-of-domain data consistently improves performance. For instance, if a researcher has 20,000 words of specific SOU data, adding 1 million words of general Brown data reduces perplexity as effectively as doubling the amount of expensive, domain-specific data. The hierarchy allows the model to learn exactly how much to trust general data for any given context, ensuring that out-of-domain data never "pollutes" the specific model.
Conclusion
The DHPYLM provides a robust, principled method for domain adaptation. While the Bayesian approach is computationally intensive due to the complexity of Gibbs sampling in a massive hierarchy, it offers superior performance and data efficiency. Future work may involve extending the model to support longer n-gram contexts and integrating it with topic models to capture broader semantic themes across domains.
Summary updated