A Privacy-Preserving Corpus for Occupational Health in Spanish: Evaluation for NER and Classification Tasks

Claudio Aracena, Luis Miranda, Thomas Vakili, Fabián Villena, Tamara Quiroga, Fredy Núñez-Torres, Victor Rocco, Jocelyn DunstanNAACL 2024

TL;DR

This paper introduces a new, publicly available, de-identified corpus of 2,869 Spanish occupational health reports. They demonstrate its utility for Named Entity Recognition (NER) of Personally Identifiable Information (PII) and for an insurance coverage classification task, showing that de-identification doesn't significantly harm downstream performance.

What Problem Does This Paper Address?

The increasing reliance on large text datasets for Natural Language Processing (NLP) models, especially powerful transformer models, raises significant privacy concerns. These models can inadvertently memorize and potentially leak sensitive information, such as Personally Identifiable Information (PII), from their training data. This is particularly problematic in sensitive domains like healthcare. While many large NLP datasets exist for English, there's a notable scarcity of similar resources for other languages, including Spanish. This lack of data hinders the development and evaluation of privacy-preserving NLP tools for these languages, especially for cross-institutional validation, which is often challenging due to data sharing restrictions.

Methods

The core of this work involves creating and evaluating a new corpus. The authors collected 2,869 medical and admission reports from an occupational insurance and health provider. These reports were then meticulously annotated for PII by two annotators, with a third researcher resolving conflicts to create a gold standard. A key aspect of their methodology is the de-identification process, where identified PII entities are replaced with their class names (e.g., __FirstName__).

They then evaluated this corpus through several experiments:

  • Few-Shot NER with GPT-3.5: They tested the ability of a large language model (GPT-3.5) to perform PII detection using only a few examples (few-shot learning), comparing its performance to human annotators.
  • NER Model Training and Evaluation: They trained and evaluated NER models (mDeBERTaV3 and IXABERTes-v2) for PII detection using both their newly created corpus and the existing MEDDOCAN corpus. This allowed them to assess the transferability of models trained on different datasets and the generalizability of PII detection models.
  • Insurance Coverage Classification: They used the corpus to train a classifier for insurance coverage decisions, comparing the performance of models trained on the original (PII-containing) corpus versus the de-identified corpus.

Datasets

  • Newly Introduced Corpus: A pseudonymous corpus of 2,869 medical and admission reports in Spanish, collected from an occupational insurance and health provider. This corpus is de-identified and publicly available.
  • MEDDOCAN: A synthetic corpus of 1,000 documents for PII identification in Spanish, used for comparison and model training.

Key Insights and Ideas

  • Cross-institutional PII detection is challenging: Models trained on one dataset (like MEDDOCAN) perform significantly worse when tested on a different, albeit related, dataset (their new corpus), highlighting the difficulty of achieving generalizable PII detection across different data sources.
  • De-identification doesn't necessarily hurt downstream performance: For the insurance coverage classification task, using the de-identified corpus resulted in performance comparable to using the original corpus, suggesting that privacy-preserving measures might not always come at a significant cost to utility.
  • The value of real-world, de-identified data: The authors emphasize that their corpus, derived from actual patient data (though de-identified), offers a more realistic benchmark than synthetic datasets like MEDDOCAN, which might contain artifacts that simplify classification.
  • GPT-3.5 shows promise but isn't state-of-the-art for NER: While GPT-3.5 performed reasonably well in few-shot PII detection, outperforming cross-institutional taggers, it was not as effective as fine-tuned models. Additionally, its output sometimes included unintended text modifications.

Key Results

  • Inter-Annotator Agreement (IAA): High IAA was achieved, with macro F₁ scores ranging from 0.86 to 0.93, indicating reliable annotation.
  • NER Performance: Fine-tuned models achieved an F₁ score of 0.911 for PII detection on their corpus, while GPT-3.5 achieved 0.720.
  • Insurance Coverage Classification: The performance difference between using the original and de-identified corpus was minimal, with F₁ scores around 0.708-0.763 for the de-identified version, suggesting that de-identification is effective without substantial loss of utility for this task.

Limitations and Future Work

  • Limitations:
    • The performance gap between models trained on MEDDOCAN and their corpus suggests that current PII detection models may not generalize well across different data sources or domains.
  • GPT-3.5's tendency to alter original text during few-shot NER poses a challenge for direct comparison with annotations.
  • Future Work:
    • Exploring the use of synthetic data to replace masked PII entities, which could further enhance privacy while enabling robust model development.
  • Investigating how to build models that are more robust to variations in PII representation across different institutions.

Summary updated

BibTeX
@inproceedings{aracena2024privacy,
  author = {Claudio Aracena and Luis Miranda and Thomas Vakili and Fabián Villena and Tamara Quiroga and Fredy Núñez-Torres and Victor Rocco and Jocelyn Dunstan},
  title = {{A Privacy-Preserving Corpus for Occupational Health in Spanish: Evaluation for NER and Classification Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure