TL;DR
This paper introduces DomainInv, a novel unsupervised domain adaptation framework for extractive Question Answering (QA). DomainInv tackles the challenge of QA systems failing on new domains by combining domain-invariant fine-tuning with adversarial label correction, achieving state-of-the-art performance without requiring labeled target domain data.
What Problem Does This Paper Address?
Machine learning models, particularly Question Answering (QA) systems, often struggle when deployed in real-world scenarios due to "domain shift." This means a model trained on one dataset (source domain) performs poorly on another dataset (target domain) because the underlying data distributions differ. Existing domain adaptation methods for QA often rely on generating synthetic data or using "pseudo-labels" (predictions on unlabeled data). However, synthetic data generation is computationally expensive, and pseudo-labeling can introduce noise and errors, especially when distinguishing correct from incorrect labels is difficult. This paper addresses the need for an unsupervised domain adaptation method that is robust to noisy pseudo-labels and effectively transfers knowledge from a labeled source domain to an unlabeled target domain for extractive QA.
Methods
DomainInv is built upon two core components:
- Domain Invariant Fine-Tuning: This technique aims to make the QA model's representations invariant to the domain. It involves introducing a "domain transformation layer" within the BERT encoder. This layer learns a "domain shift vector" that captures the differences between source and target domain representations at each layer. By fusing this shift vector with the hidden states, the model learns to represent target domain data in a way that is closer to the source domain's style, without explicit target domain labels. This process is applied iteratively.
- Adversarial Label Correction: This component addresses the noise inherent in pseudo-labels generated by the initial QA model (classifier C1). It uses an adversarial approach to identify target instances that are significantly different from the source domain. The method then aims to minimize the discrepancy between two classifiers (C1 and a target-aware classifier C2) by generating target domain features that are closer to the source domain's class support (i.e., the correct start and end positions for answers). This is framed as a min-max game, where the goal is to make the target domain features indistinguishable from the source domain features for the correct answer spans.
Datasets
The paper uses the following datasets:
- Source Domain: SQuAD v1.1 (used for training the initial QA model).
- Target Domains: MRQA Split 1, including NaturalQuestions, HotpotQA, SearchQA, TriviaQA, and NewsQA.
Key Insights and Ideas
- Noise-Robustness is Crucial: The paper highlights that relying solely on pseudo-labels for domain adaptation can be detrimental due to noise. DomainInv's adversarial label correction is designed to mitigate this by actively identifying and correcting for instances where pseudo-labels are likely to be wrong.
- Target Style Augmentation: Injecting target domain "style" into the source domain fine-tuning process (Domain Invariant Fine-Tuning) is key to adapting the model without direct target supervision.
- Class-Wise Alignment: The adversarial correction specifically focuses on aligning the predicted start and end positions of answers, recognizing that these are the critical outputs for extractive QA.
- Iterative Improvement: The framework employs an iterative process, first fine-tuning for domain invariance and then refining with adversarial label correction, suggesting a synergistic relationship between the two components.
Key Results
DomainInv demonstrates state-of-the-art performance on multiple target QA datasets. Specifically:
- On the target domains (HotpotQA, NaturalQ., NewsQA, SearchQA, TriviaQA), DomainInv consistently outperforms existing domain adaptation baselines.
- For example, on HotpotQA, DomainInv achieves an EM/F1 score of 52.92/66.71, compared to the best baseline QADA at 50.80/65.75.
- The paper also shows that DomainInv outperforms supervised training with a limited number of target domain samples (10K), indicating its efficiency in leveraging unlabeled data.
- Ablation studies show that removing the adversarial label correction component leads to a significant performance drop, underscoring its importance.
Limitations and Future Work
- Domain Adaptation Layer Assumption: The domain adaptation layer assumes equal importance for all tokens, potentially overlooking the influence of self-attention mechanisms on token distribution. Future work could explore incorporating attention-weighted representations.
- Feature Encoder Adjustment: The adversarial label correction currently adjusts the feature encoder solely based on the target domain. Future research could explore jointly aligning both source and target domains for potentially better results.
Summary updated