TALP-UPC at ProbSum 2023: Fine-tuning and Data Augmentation Strategies for NER

Neil Torrero, Gerard Sant, Carlos EscolanoACL 2023

TL;DR

This paper tackles the challenge of automatically extracting health issues from electronic medical records for the BioNLP 2023 Problem List Summarization task. The TALP-UPC team achieved this by fine-tuning BERT models and employing data augmentation techniques, demonstrating that these methods are effective even with limited domain-specific data.

What Problem Does This Paper Address?

Healthcare professionals face immense workloads, and automating repetitive tasks like summarizing patient medical records can significantly alleviate their cognitive load, leading to better patient care. The core challenge in this domain is the scarcity of annotated data. Medical records are sensitive, requiring human oversight for privacy, making large-scale annotation expensive and time-consuming. This lack of data makes it difficult to train robust Natural Language Processing (NLP) models, particularly for tasks like Named Entity Recognition (NER), which is crucial for identifying health issues.

Methods

The TALP-UPC team's approach is built upon two main pillars: fine-tuning pre-trained language models and data augmentation.

  • Fine-tuning BERT: They leverage BERT, a powerful pre-trained language model, and fine-tune it for the NER task. They explored a "two-step fine-tuning" strategy. The first step involves pre-training on broader biomedical NER datasets (NCBI Disease and BC5CDR) to learn general biomedical entity recognition. The second step then fine-tunes this model specifically on the target MIMIC dataset, adapting it to the nuances of the problem list summarization task. This two-step approach aims to improve generalization and domain adaptation.
  • Data Augmentation: To combat data scarcity and prevent overfitting, they employed data augmentation by randomly masking tokens in the input sentences. This forces the model to rely more on contextual information rather than specific words or their positions, leading to more robust representations. They experimented with different masking probabilities (5% to 40%).
  • Data Preprocessing and Post-processing: Before model training, they cleaned the MIMIC data by removing anonymization tags. For post-processing, they addressed issues like duplicate terms and inconsistent acronym usage by removing redundancies and standardizing the output format using a semicolon delimiter.

Datasets

The study utilized the following datasets:

  • MIMIC III: This is the primary dataset for the BioNLP 2023 Problem List Summarization task, containing electronic health records. It's a relatively small dataset with 608 training sentences.
  • NCBI Disease: A biomedical dataset containing disease names and concept annotations from PubMed abstracts.
  • BC5CDR: Another biomedical dataset focused on chemical-disease relations.

By combining NCBI Disease and BC5CDR with MIMIC, the total training data increased significantly to 11,269 examples, highlighting the benefit of leveraging out-of-domain biomedical data.

Key Insights and Ideas

  • Two-step fine-tuning is superior: The two-step fine-tuning approach, where the model is first trained on general biomedical NER data and then on the specific task data, consistently outperformed a single-step fine-tuning on all available data. This suggests that learning broader biomedical patterns first is beneficial.
  • Data augmentation is crucial for small datasets: Random token masking proved effective in delaying overfitting and improving performance, especially when dealing with limited domain-specific data like MIMIC.
  • Public vs. Private Test Sets: The authors observed a notable performance difference between their private test set and the public test set. They hypothesize that the public test set might be cleaner, leading to a more accurate reflection of the model's true performance.
  • Post-processing matters: Simple post-processing steps, like removing duplicates and standardizing output formats, significantly improved the evaluation metrics, demonstrating the importance of aligning model output with evaluation criteria.

Key Results

The TALP-UPC team's best-performing model, which utilized the two-step fine-tuning strategy combined with 40% data augmentation and post-processing, achieved the following:

  • Private Test Set: An F1 score improvement of 4% was observed after applying the post-processing step.
  • Public Test Set: The best model on the public test set was the one with 40% data augmentation, which was also their final submission.

Overall, the two-step fine-tuning approach outperformed fine-tuning on all available data by over 1% on both private and public test sets.

Limitations and Future Work

  • Limitations:
    • Data Scarcity: Despite data augmentation, the limited size of the MIMIC dataset remains a challenge.
  • Annotation Inconsistencies: The manual labeling process revealed inconsistencies in the original medical notes, such as the presence of additional diagnoses not explicitly in the ground truth summary, and variations in acronym usage, which impacted evaluation.
  • Ethical Considerations: The authors emphasize that the system should be used under human expert supervision due to the critical nature of medical decisions and the potential for errors, especially with partially annotated entities.
  • Future Work: The team plans to explore leveraging additional non-annotated data from the biomedical domain to further improve model performance.

Summary updated

BibTeX
@inproceedings{torrero2023talp,
  author = {Neil Torrero and Gerard Sant and Carlos Escolano},
  title = {{TALP-UPC at ProbSum 2023: Fine-tuning and Data Augmentation Strategies for NER}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure