UDALM: Unsupervised Domain Adaptation through Language Modeling

Constantinos Karouzos, Georgios Paraskevopoulos, Alexandros Potamianos

Introduction

In Natural Language Processing (NLP), models often suffer from "domain shift"—a performance drop when a model trained on a source domain (e.g., movie reviews) is applied to a target domain (e.g., kitchen appliance reviews). Unsupervised Domain Adaptation (UDA) aims to mitigate this by leveraging labeled source data and unlabeled target data. While Domain Adversarial Training (DAT) is a common approach, it is often unstable and requires extensive tuning.

UDALM (Unsupervised Domain Adaptation through Language Modeling) offers a simpler, more robust alternative. By integrating Masked Language Modeling (MLM) on the target domain directly into the fine-tuning process, UDALM achieves state-of-the-art results with significantly higher stability than adversarial methods.

Methodology

The UDALM framework consists of a three-stage pipeline designed to adapt a BERT model to a new domain without target-domain labels.

  1. General Pretraining: The process begins with a standard pretrained BERT model.
  2. Domain Pretraining: The model undergoes further pretraining on unlabeled target-domain data using the standard MLM objective.
  3. UDALM Fine-tuning: This is the critical stage where the model is trained using a mixed loss function. In each training batch, the model simultaneously minimizes classification loss on labeled source data and MLM loss on unlabeled target data.
The UDALM training pipeline: (a) General Pretraining, (b) Domain Pretraining, and (c) UDALM Fine-tuning with mixed loss.

This joint training allows the model to learn the specific linguistic nuances of the target domain while simultaneously learning the downstream task from the source domain.

Experimental Setup

The authors evaluated UDALM using the Amazon Reviews Multi-Domain Sentiment Dataset, focusing on four domains: Books (B), DVDs (D), Electronics (E), and Kitchen appliances (K). They tested 12 "source → target" pairs, using 2,000 labeled reviews for the source and approximately 17,000–20,000 unlabeled reviews for the target.

Results and Discussion

UDALM demonstrated superior performance and stability across all tested scenarios.

State-of-the-Art Performance

UDALM achieved an average accuracy of 91.74% across the 12 domain pairs, representing a 1.11% absolute improvement over previous state-of-the-art methods. As shown in the results table, UDALM consistently provides the highest accuracy in nearly every domain transfer task.

Sentiment classification accuracy (%) for 12 domain pairs. UDALM consistently outperforms baselines and adversarial methods.

Stability and Efficiency

Unlike adversarial methods (DAT), which exhibit high variance and often fail to improve upon simple baselines, UDALM is remarkably stable. It also shows high sample efficiency, providing a robust performance boost even with limited unlabeled target data. This stability is a significant advantage for real-world applications where target labels are unavailable for validation.

The A-distance Paradox

A key theoretical finding is that UDALM achieves high accuracy despite not minimizing "A-distance" (a measure of domain divergence) as aggressively as adversarial methods. This suggests that forcing feature overlap between domains can sometimes distort the feature space and degrade performance, whereas UDALM's language-modeling-based approach preserves useful domain-specific features.

Conclusion

UDALM simplifies unsupervised domain adaptation by treating it as a multi-task learning problem. By combining task-specific classification with target-domain language modeling, it provides a stable and effective way to bridge the domain gap. Future work will explore UDALM's applicability to more complex tasks like Named Entity Recognition and its potential for temporal and stylistic adaptation.

Summary updated

BibTeX
@inproceedings{karouzos2021udalm,
  author = {Constantinos Karouzos and Georgios Paraskevopoulos and Alexandros Potamianos},
  title = {{UDALM: Unsupervised Domain Adaptation through Language Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=TLN0dDwSAxA},
}
Figure