MedAI at SemEval-2021 Task 10: Negation-aware Pre-training for Source-free Negation Detection Domain Adaptation

Jinquan Sun, Qi Zhang, Yu Wang, Lei ZhangACL 2021

TL;DR

This paper tackles the challenge of adapting a negation detection model to a new clinical domain without access to the original training data (source-free domain adaptation). They achieve state-of-the-art results by introducing "Negation-aware Pre-training" (NAP) to imbue models with negation knowledge and by generating high-quality pseudo-labels for target domain data.

What Problem Does This Paper Address?

Extracting accurate information from clinical text is crucial for healthcare. Negation detection, identifying when a clinical concept is negated (e.g., "no fever"), is a key component of this. However, training robust negation detection models often requires large amounts of labeled data. In real-world scenarios, especially with sensitive patient data, accessing and sharing this labeled data across different institutions or domains is problematic due to privacy concerns. This is where "source-free unsupervised domain adaptation" comes in: the goal is to adapt a model trained on a labeled source dataset to a new, unlabeled target dataset, without ever seeing the original source data again. This paper specifically addresses this challenge for negation detection in clinical text, aiming to improve model performance on a new clinical dataset using only a pre-trained source model and unlabeled target data.

Methods

The authors propose a two-stage approach:

  1. Negation-aware Pre-training (NAP): This stage aims to infuse general negation knowledge into the model before it's adapted to the specific target domain. NAP involves three self-supervised tasks:
  2. Common Token Masking: Standard masked language modeling (similar to BERT/RoBERTa) to capture general language understanding of the target domain.
  3. Negation Cue Prediction: A token classification task where the model learns to identify negation cues (like "not," "no," "without") based on a provided lexicon.
  4. Pseudo Negation Detection: A sequence classification task that mimics the final negation detection task but uses heuristically generated pseudo-labels on target domain data. This helps the model learn complex relationships between negation cues and target mentions.
  5. Pseudo Label Training: After NAP, the model is fine-tuned on the target domain data. To overcome the lack of labeled target data, they generate "pseudo-labels." This involves:
  6. Using the source model to predict probabilities for target samples.
  7. Employing a "Confidence Threshold Search" method, inspired by self-entropy, to identify target samples with highly confident predictions from the source model. These confident predictions are then used as pseudo-labels for training.

Datasets

  • Source Domain Data: Provided by organizers as a "span-in-context" negation detection model trained on the SHARP Seed dataset (clinical notes from Mayo Clinic). This dataset contained 10,259 samples with 902 negated instances.
  • Target Domain Data: Extracted from the MIMIC-III version 1.4 dataset (de-identified health records from Beth Israel Deaconess Medical Center).
  • Approximately 50,000 pseudo samples were extracted for the NAP stage.
  • The official test dataset comprised 9,580 samples.
  • A custom test dataset was created with 500 negated and 500 non-negated samples, featuring diverse negation cues and styles, to further validate the method.

Key Insights and Ideas

  • NAP is Crucial for Negation Knowledge: Simply applying masked language modeling to the target domain (MLM) offers some improvement, but it's not enough. The NAP stage, by explicitly training on negation cues and pseudo-negated examples, significantly boosts performance, indicating that general negation knowledge transfer is vital.
  • Pseudo-labeling Quality Matters: Generating high-quality pseudo-labels is key for effective domain adaptation when labeled data is scarce. The confidence threshold search method helps filter out noisy labels, leading to better fine-tuning.
  • Source-Free Adaptation is Feasible and Effective: The paper demonstrates that achieving state-of-the-art results in a challenging source-free domain adaptation setting for negation detection is possible with the proposed techniques.

Key Results

The proposed method achieved the 1st place in the SemEval-2021 Task 10 Negation Detection Track, with an F1-score of 0.822 on the official blind test set.

  • On the official test set, their "Source + NAP" method (0.822 F1) significantly outperformed "Source" (0.685 F1) and "Source + MLM" (0.724 F1).
  • On the customized test set, the "Source + NAP" method achieved a Precision of 0.894 and a Recall of 0.833, outperforming baselines that struggled with less common negation cues and double-negatives.

Limitations and Future Work

  • Long-term Negation Dependency: The authors acknowledge that their adapted model sometimes struggles with long-term negation dependencies, where the negation cue is distant from the target mention.
  • Heuristic Rule Limitations: The pseudo-label generation for NAP relies on simple heuristic rules, which can be limiting.
  • Future Directions:
    • Develop more robust pseudo-label generation methods.
  • Incorporate additional linguistic features like part-of-speech tags and dependency relations into pre-training and training to better handle complex negation scenarios.

Summary updated

BibTeX
@inproceedings{sun2021medai,
  author = {Jinquan Sun and Qi Zhang and Yu Wang and Lei Zhang},
  title = {{MedAI at SemEval-2021 Task 10: Negation-aware Pre-training for Source-free Negation Detection Domain Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure