Abstract
In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.
Introduction
Machine learning models often suffer from domain shift: a model trained on one domain (e.g., book reviews) typically performs poorly on another (e.g., electronics reviews) because the vocabulary and distribution of features change. Unsupervised Domain Adaptation (UDA) aims to bridge this gap using labeled data from a "source" domain and only unlabeled data from a "target" domain.
While techniques like Domain Adversarial Training (DAT) are popular, they are often unstable, highly sensitive to hyperparameters, and can inadvertently distort the model's internal semantic representations. The authors of UDALM seek a simpler, more stable method to adapt high-capacity pretrained language models (LMs) to new distributions without requiring target-domain labels.
Methods: The UDALM Pipeline
The authors propose a three-stage pipeline called UDALM (Unsupervised Domain Adaptation through Language Modeling), which leverages the inherent strengths of Masked Language Modeling (MLM) to bridge domain gaps.
The pipeline consists of:
- General Pretraining: Starting with a standard pretrained model like BERT-base.
- Domain Pretraining: Continuing training on unlabeled target-domain data using the MLM objective to familiarize the model with the target vocabulary.
- Mixed-Loss Fine-tuning: This is the core innovation. During task-specific fine-tuning, the model minimizes a mixed loss: $$L = \lambda L_{CLF}(\text{source}) + (1 - \lambda) L_{MLM}(\text{target})$$ The model interleaves batches of labeled source data for classification and unlabeled target data for MLM. The weighting factor $\lambda$ is dynamically adjusted based on the ratio of source to target samples.
Experimental Setup
The method is evaluated on the Amazon Reviews Multi-Domain Sentiment Dataset, a standard UDA benchmark. The study covers four domains: Books (B), DVDs (D), Electronics (E), and Kitchen appliances (K), resulting in 12 source-target pairs. Each domain contains 2,000 labeled reviews for training/testing and up to 20,000 unlabeled reviews for adaptation.
Key Results and Robustness
UDALM achieved an average accuracy of 91.74% across all 12 domain pairs, marking a 1.11% absolute improvement over previous state-of-the-art methods such as p+CFd and BERT-DAAT. Beyond raw accuracy, the method demonstrated superior stability and sample efficiency.
As shown in the figure above, UDALM remains effective even with very limited unlabeled target data, outperforming baselines with as few as 500 target-domain samples. Furthermore, it shows much lower variance across different runs compared to adversarial approaches like DAT BERT, which can be highly erratic.
Discussion and Insights
- MLM as a Regularizer: Keeping the MLM objective active during fine-tuning acts as a powerful regularizer. It prevents the model from overfitting to source-domain-specific features (e.g., the word "plot" in book reviews) that might not be relevant or could even be misleading in the target domain (e.g., kitchen appliances).
- The Stopping Criterion Problem: In UDA, the lack of labeled target data makes validation difficult. The authors discovered that the minimum of the mixed loss serves as an excellent proxy for peak target-domain performance, providing a practical way to know when to stop training.
- A-distance vs. Accuracy: The paper challenges the common assumption that reducing "A-distance" (a measure of domain divergence) strictly correlates with better accuracy. For high-capacity models like BERT, adversarial alignment can sometimes destroy the semantic clusters necessary for the task, whereas UDALM maintains these clusters while adapting to the new domain's language.
Conclusion
UDALM provides a robust and sample-efficient framework for unsupervised domain adaptation. By integrating MLM directly into the fine-tuning process, it achieves state-of-the-art performance with greater stability than adversarial methods. Future work will extend this approach to more complex NLP tasks such as sequence labeling, question answering, and part-of-speech tagging.
Summary updated