Introduction
State-of-the-art Reading Comprehension (RC) models, such as BERT and its successors, have achieved remarkable performance by leveraging massive datasets like SQuAD. However, these models often struggle with few-shot domain adaptation. When transitioning from a resource-rich source domain (e.g., Wikipedia) to a specialized or low-resource target domain (e.g., medical texts or social media), full-model fine-tuning frequently leads to overfitting. With hundreds of millions of parameters and only a few hundred training examples, the model's excessive "degrees of freedom" allow it to memorize noise in the target data, often at the cost of the general linguistic knowledge acquired during pre-training.
This paper introduces ALTER (Adaptable Lottery), a method that leverages the Lottery Ticket Hypothesis to improve domain adaptation. The core insight is that updating fewer parameters can lead to better generalization in low-data regimes.
The ALTER Method
The ALTER framework identifies a sparse "winning ticket" subnetwork within a source model and adapts only those specific weights to the target domain. This process involves three primary stages:
- Source Training: A standard Transformer model (e.g., BERT-base) is first trained on a large source dataset, such as SQuAD v1.1.
- Identifying the "Winning Ticket": Rather than relying solely on weight magnitude pruning, the authors employ Self-Attention Attribution. By calculating which attention heads contribute most significantly to the model's predictions, they can protect parameters within critical heads while pruning less important ones. This results in a sparse subnetwork that retains the most relevant features for the task.
- Subnetwork Adaptation: The identified subnetwork is then fine-tuned on the small target domain dataset while the majority of the model's parameters remain frozen. This approach combines unstructured pruning (at the weight level) with structured insights (at the attention head level).
Key Insights: Attention Stability and Structure
A pivotal discovery in this research is the concept of Attention Stability. The authors found that attention heads identified as important in the source domain (SQuAD) remain highly relevant across diverse target domains.
As shown in the correlation analysis, a head vital for understanding syntax or entity relationships in Wikipedia articles is typically just as vital for understanding news articles or search queries. This stability suggests that the "winning ticket" for the source domain is inherently adaptable to new contexts. Furthermore, the results indicate that for small datasets, the overparameterization of large models acts as noise. By restricting the trainable space to approximately 25% of the original parameters, ALTER creates a more robust and efficient adaptation process.
Experimental Results
The researchers evaluated ALTER against five diverse target domains: NewsQA (news), TriviaQA (trivia), TweetQA (informal social media), NaturalQuestions (search queries), and QuAC (conversational QA).
- Superior Performance: ALTER outperformed full-model fine-tuning on 4 out of 5 target domains when using 1,024 training examples.
- Low-Data Robustness: The performance gap was even more pronounced in extremely low-data regimes (64 or 128 examples). For instance, on NewsQA with 128 examples, ALTER achieved an F1 score of ~58.1, significantly higher than the ~57.4 achieved by full fine-tuning.
- Efficiency: The method typically updated only 21 million parameters—roughly 25% of BERT-base's non-embedding parameters—while achieving higher accuracy than models updating all 84 million parameters.
Conclusion and Future Work
While ALTER showed exceptional results, it faced challenges with the NaturalQuestions (NQ) dataset, where it slightly trailed full fine-tuning. This suggests that some domains may require larger or more complex subnetworks to capture their specific nuances. Additionally, while the initial pruning phase adds computational overhead, the resulting efficiency during adaptation is a significant gain. Future research may explore combining ALTER with regularization techniques like Elastic Weight Consolidation (EWC) to further enhance performance and stability in few-shot learning scenarios.
Summary updated