Abstract
Domain adaptation is a critical challenge in statistical learning, where the i.i.d. assumption—that training and test data are drawn from the same distribution—is frequently violated in practice. We present the Maximum Entropy Genre Adaptation Model (MegaM), a novel framework that decomposes feature distributions into general and domain-specific components. By utilizing a three-way split—general, truly in-domain, and truly out-of-domain—MegaM leverages large source datasets to estimate shared parameters, thereby enhancing performance on target domains with limited supervision. Experimental results across multiple NLP tasks demonstrate a 26.5% average error reduction over state-of-the-art baselines.
Introduction
The "Golden Rule" of statistical learning is often impractical in real-world applications. While labeled data may be abundant for a source domain (e.g., the Wall Street Journal), building a classifier for a target domain (e.g., informal chat transcripts or biomedical text) remains difficult due to distribution mismatch. Hand-labeling sufficient data for every new domain is prohibitively expensive.
The challenge addressed here is how to utilize a large "related but different" source dataset to boost performance on a small target dataset. Previous approaches, such as using out-of-domain data as a prior, were often asymmetric and lacked a robust theoretical foundation for feature sharing. MegaM addresses this by treating domain adaptation as a latent variable problem within a discriminative framework.
Methods
MegaM posits that data points are generated from a mixture of distributions. Specifically, it distinguishes between features that are "general" ($q^{(g)}$) across all domains and those that are "truly in-domain" ($q^{(i)}$) or "truly out-of-domain" ($q^{(o)}$).
Under this framework, an in-domain data point is modeled as being drawn from either the general or the truly in-domain distribution. Conversely, an out-of-domain point is drawn from either the general or the truly out-of-domain distribution.
Algorithms and Implementation
The framework is applied to standard log-linear classifiers and Maximum Entropy Markov Models (MEMMs). Because the specific "domain" of a feature (whether it is acting as 'general' or 'domain-specific' in a given context) is a hidden variable, the authors employ Conditional Expectation Maximization (CEM). Unlike standard EM, which maximizes joint likelihood, CEM is tailored for discriminative models by focusing on the conditional probability of the labels, ensuring the model optimizes directly for classification accuracy.
Experimental Evaluation
The authors evaluated MegaM on three Natural Language Processing (NLP) tasks across four datasets: Mention Type Classification (ACE 2005), Mention Tagging (identifying entity boundaries in ASR transcripts), and Recapitalization (restoring case to lowercase text).
Key Results
MegaM consistently outperformed all baselines, including simple data pooling and prior-based adaptation. Across all tasks, the model achieved a 26.5% average reduction in error compared to the best alternative method.
In the Mention Type task, accuracy increased from 81.2% (using only in-domain data) to 92.1% with MegaM. Furthermore, learning curves demonstrated that MegaM is highly data-efficient; for instance, MegaM with just 10 in-domain examples outperformed a prior-based model using 100 examples. This efficiency is attributed to the "bridge effect," where out-of-domain data provides a high-fidelity estimate of shared parameters, allowing target data to focus exclusively on learning domain-specific nuances.
Discussion and Conclusion
MegaM provides a robust, symmetric framework for domain adaptation. Unlike prior-based models, MegaM treats both datasets as equally informative about the general distribution. While the model's computational complexity is higher—training takes approximately 15 times longer than standard models due to CEM iterations—its ability to effectively "bridge" domains makes it a powerful tool for low-resource settings. Future research will explore its application to Conditional Random Fields (CRFs) and its synergy with active learning strategies.
Summary updated