Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension

Haichao Zhu, Zekun Wang, Heng Zhang, Ming Liu, Sendong Zhao, Bing Qin

Introduction

State-of-the-art Reading Comprehension (RC) models, such as BERT and its successors, have achieved remarkable performance by leveraging massive datasets like SQuAD. However, these models often struggle with few-shot domain adaptation. When transitioning from a resource-rich source domain (e.g., Wikipedia) to a specialized or low-resource target domain (e.g., medical texts or social media), full-model fine-tuning frequently leads to overfitting. With hundreds of millions of parameters and only a few hundred training examples, the model's excessive "degrees of freedom" allow it to memorize noise in the target data, often at the cost of the general linguistic knowledge acquired during pre-training.

This paper introduces ALTER (Adaptable Lottery), a method that leverages the Lottery Ticket Hypothesis to improve domain adaptation. The core insight is that updating fewer parameters can lead to better generalization in low-data regimes.

The ALTER Method

The ALTER framework identifies a sparse "winning ticket" subnetwork within a source model and adapts only those specific weights to the target domain. This process involves three primary stages:

The ALTER workflow: identifying and adapting a sparse subnetwork.
  1. Source Training: A standard Transformer model (e.g., BERT-base) is first trained on a large source dataset, such as SQuAD v1.1.
  2. Identifying the "Winning Ticket": Rather than relying solely on weight magnitude pruning, the authors employ Self-Attention Attribution. By calculating which attention heads contribute most significantly to the model's predictions, they can protect parameters within critical heads while pruning less important ones. This results in a sparse subnetwork that retains the most relevant features for the task.
  3. Subnetwork Adaptation: The identified subnetwork is then fine-tuned on the small target domain dataset while the majority of the model's parameters remain frozen. This approach combines unstructured pruning (at the weight level) with structured insights (at the attention head level).

Key Insights: Attention Stability and Structure

A pivotal discovery in this research is the concept of Attention Stability. The authors found that attention heads identified as important in the source domain (SQuAD) remain highly relevant across diverse target domains.

Correlation of attention head importance between NewsQA and SQuAD.

As shown in the correlation analysis, a head vital for understanding syntax or entity relationships in Wikipedia articles is typically just as vital for understanding news articles or search queries. This stability suggests that the "winning ticket" for the source domain is inherently adaptable to new contexts. Furthermore, the results indicate that for small datasets, the overparameterization of large models acts as noise. By restricting the trainable space to approximately 25% of the original parameters, ALTER creates a more robust and efficient adaptation process.

Experimental Results

The researchers evaluated ALTER against five diverse target domains: NewsQA (news), TriviaQA (trivia), TweetQA (informal social media), NaturalQuestions (search queries), and QuAC (conversational QA).

  • Superior Performance: ALTER outperformed full-model fine-tuning on 4 out of 5 target domains when using 1,024 training examples.
  • Low-Data Robustness: The performance gap was even more pronounced in extremely low-data regimes (64 or 128 examples). For instance, on NewsQA with 128 examples, ALTER achieved an F1 score of ~58.1, significantly higher than the ~57.4 achieved by full fine-tuning.
  • Efficiency: The method typically updated only 21 million parameters—roughly 25% of BERT-base's non-embedding parameters—while achieving higher accuracy than models updating all 84 million parameters.

Conclusion and Future Work

While ALTER showed exceptional results, it faced challenges with the NaturalQuestions (NQ) dataset, where it slightly trailed full fine-tuning. This suggests that some domains may require larger or more complex subnetworks to capture their specific nuances. Additionally, while the initial pruning phase adds computational overhead, the resulting efficiency during adaptation is a significant gain. Future research may explore combining ALTER with regularization techniques like Elastic Weight Consolidation (EWC) to further enhance performance and stability in few-shot learning scenarios.

Summary updated

Abstract

In this paper, we propose a simple few-shot domain adaptation paradigm for reading comprehension. We first identify the lottery subnetwork structure within the Transformer-based source domain model via gradual magnitude pruning. Then, we only fine-tune the lottery subnetwork, a small fraction of the whole parameters, on the annotated target domain data for adaptation. To obtain more adaptable subnetworks, we introduce self-attention attribution to weigh parameters, beyond simply pruning the smallest magnitude parameters, which can be seen as combining structured pruning and unstructured magnitude pruning softly. Experimental results show that our method outperforms the full model fine-tuning adaptation on four out of five domains when only a small amount of annotated data available for adaptation. Moreover, introducing self-attention attribution reserves more parameters for important attention heads in the lottery subnetwork and improves the target domain model performance. Our further analyses reveal that, besides exploiting fewer parameters, the choice of subnetworks is critical to the effectiveness.
BibTeX
@inproceedings{zhu2021less,
  author = {Haichao Zhu and Zekun Wang and Heng Zhang and Ming Liu and Sendong Zhao and Bing Qin},
  title = {{Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.95},
}
Figure