MediFact at MEDIQA-CORR 2024: Why AI Needs a Human Touch

Nadia SaeedNAACL 2024

TL;DR

This paper introduces MediFact-CORR QA, a novel two-stage approach for automatically correcting single-word errors in clinical text. By combining weakly-supervised error detection with a hybrid of extractive and abstractive question-answering, the method achieves strong performance on the MEDIQA-CORR 2024 task, demonstrating the value of domain-specific feature engineering and a human-centric strategy for AI in healthcare.

What Problem Does This Paper Address?

Accurate interpretation of medical information is paramount for patient safety, but artificial intelligence (AI), particularly Large Language Models (LLMs), struggles with error-free clinical text interpretation. Challenges include linguistic ambiguities (synonyms, homonyms, polysemy), imbalanced data distribution (rare diseases being underrepresented), and ethical concerns surrounding sensitive patient data. Existing LLM approaches often focus on basic typos and grammatical errors, falling short of addressing more complex issues like patient hallucinations that can lead to misdiagnosis. Furthermore, generic LLMs trained on broad datasets may lack the domain-specific knowledge needed to capture the nuances of medical terminology and rare disease entities. This paper tackles the specific problem of correcting single-word errors in clinical notes, aiming to improve the reliability and accuracy of AI in this critical domain.

Methods

The MediFact-CORR QA approach is a two-stage process designed to be data-efficient.

  • Stage 1: Error Sentence Identification with Weak Supervision: This stage employs weakly-supervised learning to identify sentences containing errors. It utilizes Support Vector Machines (SVMs) trained on paired paragraphs (one with an error, one corrected) to discriminate between accurate and erroneous sentences. This method leverages the inherent information within error-correction pairs without requiring extensive manual annotation of every error.
  • Stage 2: Error Correction:
    • Extractive QA: For sentences identified as erroneous that have a corresponding corrected version in the training data, an extractive question-answering (QA) methodology is used. This involves fuzzy matching between the erroneous and corrected sentences to locate the most probable correct version. This approach is data-efficient as it directly extracts the answer from the provided text.
  • Abstractive QA: For erroneous sentences that do not have a direct match in the training data, a pre-trained abstractive QA model (specifically, a BART model tailored for unanswerable questions) is employed. This model analyzes contextual relationships within the erroneous sentence to generate potential corrections, enhancing robustness and generalizability for unseen errors.

The framework emphasizes domain-specific feature engineering, integrating expertise to improve accuracy.

Datasets

The primary dataset used is from the MEDIQA-CORR 2024 shared task (Ben Abacha et al., 2024a, 2024b). This dataset consists of clinical texts from MS and UW collections.

  • Training set: Comprised 2,189 texts from the MS collection.
  • Validation sets: Contained 574 texts from MS and 160 texts from UW.
  • Testing set: Comprised 925 texts from MS and UW collections.

Each text in the dataset includes split sentences, error sentence indices, and corresponding corrected sentences, along with an error flag. The shared task itself involves three challenging sub-tasks: error flag prediction, error sentence detection, and generating the correct sentence.

Key Insights and Ideas

  • Hybrid QA for Robustness: Combining extractive QA for observed errors with abstractive QA for unseen errors provides a powerful and data-efficient solution for clinical text correction.
  • Domain-Specific Feature Engineering is Crucial: Unlike generic LLMs, tailoring features to the medical context (e.g., recognizing medical terms, abbreviations, and anatomical locations) significantly improves performance. This allows the model to leverage domain knowledge.
  • Weak Supervision for Data Efficiency: The use of weakly-supervised learning for error sentence identification reduces the reliance on extensive, manually annotated datasets, which are costly and time-consuming to create in the medical domain.
  • SVMs Offer Interpretability: While LLMs are powerful, SVMs with TF-IDF features offer valuable insights through feature weights, contributing to a more interpretable model, which is vital for trust in healthcare AI.
  • Human-Centric Strategy: The paper advocates for a human-centric strategy, suggesting that AI should assist human reviewers, potentially improving efficiency and accuracy in medical record correction.

Key Results

The MediFact-CORR QA model achieved competitive results in the MEDIQA-CORR 2024 shared task:

  • Error Flags Accuracy: Secured the 2nd rank out of 106 participants.
  • Error Sentence Detection Accuracy: Attained the 8th rank.
  • Aggregate Score for NLG: Achieved the 14th rank.

Table 1 (submitted at the competition) shows specific performance metrics: R1F (0.454), BERT (0.444), BLEURT (0.439), AggScore (0.446), AggC (0.535), Error Flag (0.737), and Error Sentence (0.600). The paper also highlights that using a QA model instead of a static correction model (SVM) resulted in an improved R1F score from 0.342 to 0.454.

Limitations and Future Work

Limitations:

  • Dataset Issues: The provided dataset had pre-defined sentence indices that deviated from standard splitting, posing a challenge. The authors addressed this by comparing detected errors' content with available sentences and using the index of the most similar sentence.
  • Data Scarcity: A significant hurdle for LLMs in the medical domain is data scarcity, which limits the use of separate validation sets.

Future Work:

  • LLM Exploration: Further exploration of LLMs for pathogen identification and error correction is planned, potentially using techniques like data augmentation and transfer learning from pre-trained medical LLMs.
  • Ethical Considerations: Mitigating biases in training data and ensuring interpretability through mechanisms like attention is crucial for trust and acceptance.
  • Practical Implementation: Developing computationally efficient LLM architectures or task-specific models for pathogen identification is necessary. Continuous evaluation through active learning and performance monitoring will be vital for maintaining a robust, ethical, and interpretable system.

Summary updated

BibTeX
@inproceedings{saeed2024medifact,
  author = {Nadia Saeed},
  title = {{MediFact at MEDIQA-CORR 2024: Why AI Needs a Human Touch}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure