Automatic Medical Text Simplification: Challenges of Data Quality and Curation

Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, Sohyeong Kim, Qian YangAAAI 2021 · OpenReview

Introduction: The Health Literacy Gap

Health literacy is a fundamental determinant of patient outcomes. When medical information is inaccessible due to complex jargon and dense syntax, patients are less likely to adhere to treatment protocols. Automatic Text Simplification (ATS) offers a potential solution, yet the medical domain presents unique challenges that distinguish it from general-purpose simplification tasks. This paper investigates the current state of medical ATS, highlighting critical bottlenecks in data quality and the limitations of non-expert evaluation.

Methodology

The researchers conducted a multi-faceted audit of existing medical corpora using automated quality assessments and human pilot studies. Automated metrics included:

  • Acceptability: Evaluated via RoBERTa-based classifiers to ensure grammatical correctness.
  • Readability: Measured using standard indices such as Flesch-Kincaid Grade Level and the Automated Readability Index (ARI).
  • Diversity: Analyzed through lexical complexity and transformation types (additions, deletions, and sentence splitting) using the EASSE library.
  • Representativeness: Assessed the coverage of medical sub-domains and PICO (Patient, Intervention, Comparison, Outcome) elements.

Comparative Analysis of Medical Corpora

The study focused on two primary datasets: SIMPWIKI, a parallel corpus derived from Wikipedia and Simple Wikipedia, and MSD (Merck Manuals), a large corpus containing professional and consumer versions of medical texts.

Analysis of the transformation ratios reveals significant differences in how these datasets approach simplification. As shown in Figure 1, the MSD dataset exhibits a higher proportion of additions compared to SIMPWIKI. This suggests that the Merck Manuals rely more heavily on "elaborations"—the inclusion of definitions or context for difficult terms—rather than simple word substitution.

Comparison of transformation metrics and layman-to-expert ratios between MSD and SIMPWIKI.

The Readability and Acceptability Gap

A key finding of the research is the persistent complexity of even "simplified" medical text. While general ATS aims for a Grade 8 reading level, medical layman versions often remain significantly higher.

Readability, acceptability, and complexity metrics across expert and layman versions of medical datasets.

As detailed in Figure 2, expert texts in the MSD dataset require a Grade 15 reading level, while the layman versions only reduce this to Grade 12.6. Similarly, SIMPWIKI's layman versions hover around Grade 11.9. Furthermore, the "Acceptability" scores indicate that while SIMPWIKI maintains high grammatical standards (>97%), the MSD dataset shows lower scores (~90%), likely due to the challenges of aligning non-parallel professional and consumer content.

The Human Evaluation Challenge

The study identified an "Expert Blind Spot" in human evaluation. Laypeople often find the "expert" source text so dense that they cannot accurately judge whether a simplification preserves the original meaning. This leads to low inter-annotator agreement (Krippendorff’s alpha of 0.299) and suggests that "simplicity" is highly subjective, contingent on the reader's prior knowledge.

Conclusion: Toward Expert-Layman Collaboration

The authors argue that the field cannot rely solely on scaling raw data to improve medical ATS. Instead, they propose a hybrid "Expert-Layman" curation framework:

  1. Experts ensure content preservation and identify core concepts.
  2. Laypeople or Medical Writers refine the language for maximum accessibility.
  3. Hybrid Workflows utilize automated tools (e.g., concept extraction) to bridge the knowledge gap for human annotators.

By focusing on high-quality, expert-validated parallel corpora, the medical ATS community can move toward tools that truly improve patient comprehension and health outcomes.

Summary updated

BibTeX
@inproceedings{basu2021automatic,
  author = {Chandrayee Basu and Rosni Vasu and Michihiro Yasunaga and Sohyeong Kim and Qian Yang},
  title = {{Automatic Medical Text Simplification: Challenges of Data Quality and Curation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=DO2iicLeoF},
}
Figure