TL;DR
This paper tackles the challenge of "source-free domain adaptation," where models trained on one dataset must perform well on another without access to the original training data. The authors propose two novel unsupervised frameworks, one for negation detection and another for time expression recognition, achieving top rankings in the SemEval-2021 Task 10.
What Problem Does This Paper Address?
Traditional domain adaptation methods often assume that both labeled source data and unlabeled target data are available during training. However, this is frequently impractical due to data privacy concerns (e.g., clinical notes, social media posts) or logistical constraints. SemEval-2021 Task 10 specifically addresses this "source-free" scenario, requiring models to adapt to a new domain using only unlabeled target data and a pre-trained source model. This paper focuses on two subtasks: negation detection in clinical text and time expression recognition.
Methods
The authors developed two distinct unsupervised frameworks tailored to each subtask:
- Subtask A: Negation Detection (Text Classification)
- Pseudo-Labeling with Adaptive Prototype Memory (APM): This approach leverages pseudo-labeling, where the model generates its own labels for unlabeled target data. To improve reliability, they introduce an APM module that stores "prototypes" representing each class. Pseudo-labels are generated with high confidence by measuring the similarity of target samples to these prototypes. The model is then fine-tuned using these high-confidence pseudo-labels.
- Ensembling: To enhance robustness, five models with varying hyperparameters were trained and their predictions ensembled using a voting mechanism.
- Subtask B: Time Expression Recognition (Sequence Labeling)
- Teacher-Student Framework with Mean Teacher: This method employs a teacher-student architecture. The "teacher" model, initialized from the "student" model, generates "soft labels" (probability distributions over classes) for unlabeled target data. To further refine these labels, only tokens with high prediction confidence are selected. The "student" model is then trained to mimic these high-confidence soft labels.
- Mean Teacher: This technique further improves the teacher model's performance by updating its weights as a moving average of the student model's weights.
- Manual Rules: A set of handcrafted rules were developed based on frequently occurring feature words to improve performance.
- Ensembling: Similar to Subtask A, two ensembled models were used for the final prediction.
Datasets
- Subtask A:
- Training: SHARP Seed dataset (902 negated instances).
- Development: i2b2 2010 Challenge Dataset (5545 instances with negation status).
- Test: MIMIC III corpus v1.4.
- Note: The authors augmented their training data by including development and test data to improve fine-tuning.
- Subtask B:
- Training: Provided training set.
- Development: Provided development set.
- Test: Provided test set.
- Note: The paper mentions that category labels in Subtask B were severely imbalanced and that the dev and test sets had different distributions of time expression types.
Key Insights and Ideas
- Pseudo-labeling is powerful, but confidence matters: The success of pseudo-labeling hinges on the quality of the generated labels. By focusing on high-confidence predictions (using APM for Subtask A and confidence thresholds for Subtask B), the authors significantly improved adaptation.
- Teacher-student models can learn effectively from unlabeled data: The Mean Teacher approach, combined with soft labels and confidence filtering, demonstrates a robust way to transfer knowledge from a teacher to a student model without explicit target labels.
- Ensembling is a reliable way to boost performance: Combining multiple models, even those trained with slightly different configurations, consistently improved results across both subtasks, highlighting its value in reducing variance.
- Source-free adaptation is feasible and effective: The paper validates that achieving high performance in domain adaptation is possible even when the source data is completely unavailable during the adaptation phase.
Key Results
- Subtask A: The proposed SFDA+ensemble framework achieved the third highest F1 score among all systems. Ablation studies showed that both the APM module and the use of Lsource (loss from the source classifier) contributed to improved performance.
- Subtask B: The MT+soft+rules (ensemble) model achieved the highest F1 score among all systems. Ablation studies indicated that soft labels were crucial, with MT and manual rules providing marginal improvements. The ensemble model further boosted performance.
Limitations and Future Work
- Limitations:
- The paper acknowledges that for Subtask B, the performance on the test set dropped by an average of 2% compared to the development set, suggesting potential domain shift challenges.
- Manual error analysis revealed instances where both models struggled with long-distance keywords, parallel phrases, or complex temporal expressions.
- Future Work:
- The authors suggest exploring adversarial training and more data augmentation techniques to further enhance source-free domain adaptation.
- They also propose incorporating handcrafted rules to address specific error patterns identified in the analysis.
Summary updated