Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts

Saadullah Amin, Noon Pokaratsiri Goldstein, Morgan Wixted, Alejandro Garcia-Rudolph, Catalina Martínez-Costa, Guenter NeumannACL 2022

TL;DR

This paper tackles the challenge of de-identifying clinical notes that are written in a mix of Spanish and Catalan, a common scenario in multilingual healthcare settings. They demonstrate that pre-trained language models like mBERT can effectively transfer knowledge across languages with very few examples, achieving high accuracy in de-identifying these complex texts.

What Problem Does This Paper Address?

In the digital age, healthcare systems generate vast amounts of unstructured clinical text. This data is invaluable for research and improving patient care, but it also contains sensitive Protected Health Information (PHI) that must be removed to comply with privacy regulations like GDPR. Traditional de-identification methods often rely on large, meticulously annotated datasets, which are expensive and time-consuming to create, especially for low-resource languages or code-mixed text. This paper specifically addresses the problem of de-identifying clinical notes from the stroke domain that are written in a mix of Spanish and Catalan. This code-mixed nature, combined with the limited availability of annotated data in this specific domain and language combination, presents a significant challenge for existing de-identification tools.

Methods

The core of the paper's approach is leveraging the "few-shot cross-lingual transfer" capability of pre-trained multilingual language models (LMs), specifically mBERT. The authors frame de-identification as a Named Entity Recognition (NER) task, where the goal is to identify and label PHI entities within the text.

Here's a breakdown of their methodology:

  • Few-Shot Learning: Instead of requiring thousands of labeled examples, they focus on training with a very small number of annotated samples (hundreds). This is crucial for low-resource scenarios.
  • Cross-Lingual Transfer: They exploit mBERT's ability to understand and process multiple languages. The model is pre-trained on a large English corpus (MEDDOCAN) and then fine-tuned on a small, code-mixed Spanish-Catalan dataset.
  • Multi-Task Learning: They employ a multi-task learning strategy where the model is jointly fine-tuned on both the source (MEDDOCAN) and the target (few-shot Spanish-Catalan) corpora. This helps the model generalize better and share parameters.
  • Sequential Fine-Tuning: After the initial multi-task learning, they further fine-tune the model exclusively on the few-shot target corpus to optimize performance for the specific code-mixed domain.

Datasets

  • Source Dataset: MEDDOCAN corpus (Marimon et al., 2019). This is a synthetically generated Spanish clinical notes dataset, used for pre-training and initial fine-tuning. It contains 1000 clinical case studies.
  • Target Dataset: A newly annotated dataset of Spanish-Catalan code-mixed clinical notes from the stroke domain. This dataset was created by the authors and consists of:
  • A gold evaluation set.
  • A few-shot training corpus (384 sentences).
  • An out-of-sample test set (100 notes) for generalizability evaluation.

The target dataset focuses on coarse-grained PHI categories: DATE, AGE, LOCATION, NAME, CONTACT, PROFESSION, and ID.

Key Insights and Ideas

  • mBERT's Few-Shot Cross-Lingual Power: The paper strongly demonstrates that mBERT, even with just a few hundred labeled examples, can achieve remarkable performance in cross-lingual transfer for NER, significantly outperforming models trained from scratch. This is a key enabler for de-identifying data in low-resource languages.
  • Code-Mixed Data is Tractable: The success in de-identifying Spanish-Catalan code-mixed text suggests that current multilingual models are robust enough to handle linguistic mixing, a common phenomenon in real-world clinical settings.
  • Multi-Task Learning as Regularization: Jointly training on source and target data not only helps transfer knowledge but also acts as a form of regularization, preventing overfitting on the small target dataset.
  • Domain-Specific Fine-Tuning is Crucial: While cross-lingual transfer is powerful, an additional fine-tuning step on the target domain data further boosts performance, indicating the importance of domain adaptation.
  • DATE and AGE are Relatively Easy: The model performs exceptionally well on DATE and AGE entities, likely due to their consistent representation across languages and simple numerical formats.
  • ID and PROFESSION are Challenging: The ID entity is difficult due to its low sample size in the few-shot corpus, and PROFESSION presents challenges due to ambiguity and potential mislabeling, especially in Catalan.

Key Results

  • The zero-shot F1-score on the gold evaluation set was improved from 73.7% to 91.2% by adapting mBERT from the MEDDOCAN corpus using their few-shot cross-lingual strategy.
  • The best model achieved a human-evaluated F1-score of 97.2% on an out-of-sample test set, demonstrating strong generalizability.
  • Multi-task learning improved the F1-score to 89.5%, and further fine-tuning on the few-shot target corpus boosted it to 91.2%.
  • Per-PHI-label scores showed high performance for DATE (F1 ~97%) and AGE (F1 ~92%), while ID had a very low F1 (~7%).

Limitations and Future Work

  • Dataset Release: The authors cannot release their annotated dataset due to the presence of real PHI. They plan to explore replacing real PHI with synthetic data for a GDPR-compliant release in the future.
  • Limited Language Pairs: The study focused on Spanish-Catalan. Future work could explore a wider range of language pairs and domains.
  • ID and PROFESSION Challenges: Further research is needed to improve performance on challenging entities like ID and PROFESSION, potentially through more sophisticated annotation strategies or model architectures.
  • Out-of-Sample Generalizability: While the out-of-sample results are impressive, continued evaluation on diverse, unseen datasets is important to fully understand the model's robustness.

Summary updated

BibTeX
@inproceedings{amin2022few,
  author = {Saadullah Amin and Noon Pokaratsiri Goldstein and Morgan Wixted and Alejandro Garcia-Rudolph and Catalina Martínez-Costa and Guenter Neumann},
  title = {{Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure