TL;DR
This paper tackles the challenge of adapting a negation detection model to a new clinical domain without access to the original training data (source-free domain adaptation). They achieve state-of-the-art results by introducing "Negation-aware Pre-training" (NAP) to imbue models with negation knowledge and by generating high-quality pseudo-labels for target domain data.
What Problem Does This Paper Address?
Extracting accurate information from clinical text is crucial for healthcare. Negation detection, identifying when a clinical concept is negated (e.g., "no fever"), is a key component of this. However, training robust negation detection models often requires large amounts of labeled data. In real-world scenarios, especially with sensitive patient data, accessing and sharing this labeled data across different institutions or domains is problematic due to privacy concerns. This is where "source-free unsupervised domain adaptation" comes in: the goal is to adapt a model trained on a labeled source dataset to a new, unlabeled target dataset, without ever seeing the original source data again. This paper specifically addresses this challenge for negation detection in clinical text, aiming to improve model performance on a new clinical dataset using only a pre-trained source model and unlabeled target data.
Methods
The authors propose a two-stage approach:
- Negation-aware Pre-training (NAP): This stage aims to infuse general negation knowledge into the model before it's adapted to the specific target domain. NAP involves three self-supervised tasks:
- Common Token Masking: Standard masked language modeling (similar to BERT/RoBERTa) to capture general language understanding of the target domain.
- Negation Cue Prediction: A token classification task where the model learns to identify negation cues (like "not," "no," "without") based on a provided lexicon.
- Pseudo Negation Detection: A sequence classification task that mimics the final negation detection task but uses heuristically generated pseudo-labels on target domain data. This helps the model learn complex relationships between negation cues and target mentions.
- Pseudo Label Training: After NAP, the model is fine-tuned on the target domain data. To overcome the lack of labeled target data, they generate "pseudo-labels." This involves:
- Using the source model to predict probabilities for target samples.
- Employing a "Confidence Threshold Search" method, inspired by self-entropy, to identify target samples with highly confident predictions from the source model. These confident predictions are then used as pseudo-labels for training.
Datasets
- Source Domain Data: Provided by organizers as a "span-in-context" negation detection model trained on the SHARP Seed dataset (clinical notes from Mayo Clinic). This dataset contained 10,259 samples with 902 negated instances.
- Target Domain Data: Extracted from the MIMIC-III version 1.4 dataset (de-identified health records from Beth Israel Deaconess Medical Center).
- Approximately 50,000 pseudo samples were extracted for the NAP stage.
- The official test dataset comprised 9,580 samples.
- A custom test dataset was created with 500 negated and 500 non-negated samples, featuring diverse negation cues and styles, to further validate the method.
Key Insights and Ideas
- NAP is Crucial for Negation Knowledge: Simply applying masked language modeling to the target domain (MLM) offers some improvement, but it's not enough. The NAP stage, by explicitly training on negation cues and pseudo-negated examples, significantly boosts performance, indicating that general negation knowledge transfer is vital.
- Pseudo-labeling Quality Matters: Generating high-quality pseudo-labels is key for effective domain adaptation when labeled data is scarce. The confidence threshold search method helps filter out noisy labels, leading to better fine-tuning.
- Source-Free Adaptation is Feasible and Effective: The paper demonstrates that achieving state-of-the-art results in a challenging source-free domain adaptation setting for negation detection is possible with the proposed techniques.
Key Results
The proposed method achieved the 1st place in the SemEval-2021 Task 10 Negation Detection Track, with an F1-score of 0.822 on the official blind test set.
- On the official test set, their "Source + NAP" method (0.822 F1) significantly outperformed "Source" (0.685 F1) and "Source + MLM" (0.724 F1).
- On the customized test set, the "Source + NAP" method achieved a Precision of 0.894 and a Recall of 0.833, outperforming baselines that struggled with less common negation cues and double-negatives.
Limitations and Future Work
- Long-term Negation Dependency: The authors acknowledge that their adapted model sometimes struggles with long-term negation dependencies, where the negation cue is distant from the target mention.
- Heuristic Rule Limitations: The pseudo-label generation for NAP relies on simple heuristic rules, which can be limiting.
- Future Directions:
- Develop more robust pseudo-label generation methods.
- Incorporate additional linguistic features like part-of-speech tags and dependency relations into pre-training and training to better handle complex negation scenarios.
Summary updated