DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs

Divya Jyoti Bajpai, Manjesh Kumar HanawalarXiv 2024 · DOI · arXiv

Introduction

Pre-trained Language Models (PLMs) such as BERT and RoBERTa have become the standard for Natural Language Processing (NLP) tasks. However, their significant computational requirements pose challenges for real-time applications and resource-constrained environments. Early Exit (EE) strategies have emerged as a solution, attaching intermediate classifiers to internal layers to allow "easy" samples to exit the model early, thereby reducing inference latency.

Despite their efficiency, EE models struggle with Domain Shift. When an EE model trained on a source domain (e.g., movie reviews) is applied to a target domain (e.g., medical literature), the intermediate classifiers often fail because they have not been exposed to the target distribution. Traditional Unsupervised Domain Adaptation (UDA) techniques typically focus on the final output layer, leaving the intermediate exits unoptimized. DAdEE (Unsupervised Domain Adaptation in Early Exit PLMs) addresses this gap by ensuring that every internal layer is domain-invariant, maintaining both speed and accuracy across disparate datasets.

Methodology

DAdEE employs a three-stage workflow designed to bridge the domain gap across the entire architecture of the PLM:

1. Supervised Source Training

The process begins with standard supervised learning on a labeled source dataset. During this phase, classifiers are attached to every layer of the model, and the backbone is trained to minimize prediction error across all exits.

2. Multi-level Adversarial Adaptation

The core innovation of DAdEE is its "bottom-up" alignment. Unlike standard UDA, which aligns only the final representation, DAdEE utilizes a Generative Adversarial Network (GAN) approach at every internal layer. A domain discriminator is assigned to each layer to distinguish between source and target features. Simultaneously, the model’s backbone is trained to produce features that "fool" these discriminators. This adversarial process forces the model to learn representations that are indistinguishable between domains at every stage of processing.

3. Knowledge Distillation (KD)

To mitigate the risk of "catastrophic forgetting"—where the model loses its task-specific performance while trying to adapt to a new domain—DAdEE incorporates Knowledge Distillation. The target-adapted model is encouraged to mimic the logic and probability distributions of the original source-trained model at every layer, acting as a vital regularizer.

Inference and Exit Logic

During inference, the model utilizes a confidence threshold ($\alpha$). If an intermediate classifier's prediction exceeds this threshold, the model exits immediately. If not, the sample proceeds to the next layer. This allows the model to dynamically balance speed and precision based on the complexity of the input.

Experimental Evaluation

The researchers evaluated DAdEE across two primary benchmarks:

  • Sentiment Analysis: 30 adaptation tasks using the Amazon Reviews benchmark (covering categories like Books, DVDs, Electronics, and Kitchen) and IMDB.
  • Natural Language Inference (NLI): Datasets from GLUE and ELUE, including MNLI, SNLI, RTE, QQP, MRPC, and SciTail.

Key Results

  • Accuracy and Speed: DAdEE achieved an average accuracy improvement of 2.9% over standard models and previous UDA methods like DANN. Simultaneously, it provided an average 1.61x speedup compared to vanilla BERT/RoBERTa, with some tasks reaching a 2.11x speedup.
  • Feature Alignment: t-SNE visualizations and A-distance metrics (a measure of domain divergence) confirmed that DAdEE creates tighter, more distinct clusters in the target domain than its predecessors.
  • Data Efficiency: DAdEE demonstrated remarkable scalability. Even when the labeled source dataset was reduced to only 500 samples, the framework remained robust, whereas competing state-of-the-art methods experienced significant performance degradation.

Discussion and Insights

One of the paper's most significant findings is that for efficient models, alignment must be comprehensive. By aligning features from the first layer to the last, DAdEE prevents the accumulation of domain-specific noise. Furthermore, the researchers noted that early exits can actually prevent "overthinking," where deeper layers might overfit to complex features in easy samples, leading to errors that early layers would have avoided.

However, the framework is not without limitations. The model is highly sensitive to the Knowledge Distillation process; removing KD from even a few layers can lead to noise accumulation. Additionally, the current use of static exit thresholds ($\alpha$) derived from the source domain may not be optimal for all target domains.

Conclusion

DAdEE represents a significant step forward in making efficient language models more versatile. By combining multi-level adversarial training with knowledge distillation, it ensures that the benefits of Early Exit strategies—namely high speed and reduced computation—are preserved even when models are deployed in entirely new environments. Future work may explore adaptive thresholding to further optimize the speed-accuracy trade-off in real-time applications.

Summary updated

Abstract

Pre-trained Language Models (PLMs) exhibit good accuracy and generalization ability across various tasks using self-supervision, but their large size results in high inference latency. Early Exit (EE) strategies handle the issue by allowing the samples to exit from classifiers attached to the intermediary layers, but they do not generalize well, as exit classifiers can be sensitive to domain changes. To address this, we propose Unsupervised Domain Adaptation in EE framework (DADEE) that employs multi-level adaptation using knowledge distillation. DADEE utilizes GAN-based adversarial adaptation at each layer to achieve domain-invariant representations, reducing the domain gap between the source and target domain across all layers. The attached exits not only speed up inference but also enhance domain adaptation by reducing catastrophic forgetting and mode collapse, making it more suitable for real-world scenarios. Experiments on tasks such as sentiment analysis, entailment classification, and natural language inference demonstrate that DADEE consistently outperforms not only early exit methods but also various domain adaptation methods under domain shift scenarios. The anonymized source code is available at https://github.com/Div290/DAdEE.
BibTeX
@misc{bajpai2024dadee,
  author = {Divya Jyoti Bajpai and Manjesh Kumar Hanawal},
  title = {{DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs}},
  year = {2024},
  eprint = {2410.04424},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.04424},
}
Figure