Introduction: The Health Literacy Gap
Health literacy is a fundamental determinant of patient outcomes. When medical information is inaccessible due to complex jargon and dense syntax, patients are less likely to adhere to treatment protocols. Automatic Text Simplification (ATS) offers a potential solution, yet the medical domain presents unique challenges that distinguish it from general-purpose simplification tasks. This paper investigates the current state of medical ATS, highlighting critical bottlenecks in data quality and the limitations of non-expert evaluation.
Methodology
The researchers conducted a multi-faceted audit of existing medical corpora using automated quality assessments and human pilot studies. Automated metrics included:
- Acceptability: Evaluated via RoBERTa-based classifiers to ensure grammatical correctness.
- Readability: Measured using standard indices such as Flesch-Kincaid Grade Level and the Automated Readability Index (ARI).
- Diversity: Analyzed through lexical complexity and transformation types (additions, deletions, and sentence splitting) using the EASSE library.
- Representativeness: Assessed the coverage of medical sub-domains and PICO (Patient, Intervention, Comparison, Outcome) elements.
Comparative Analysis of Medical Corpora
The study focused on two primary datasets: SIMPWIKI, a parallel corpus derived from Wikipedia and Simple Wikipedia, and MSD (Merck Manuals), a large corpus containing professional and consumer versions of medical texts.
Analysis of the transformation ratios reveals significant differences in how these datasets approach simplification. As shown in Figure 1, the MSD dataset exhibits a higher proportion of additions compared to SIMPWIKI. This suggests that the Merck Manuals rely more heavily on "elaborations"—the inclusion of definitions or context for difficult terms—rather than simple word substitution.
The Readability and Acceptability Gap
A key finding of the research is the persistent complexity of even "simplified" medical text. While general ATS aims for a Grade 8 reading level, medical layman versions often remain significantly higher.
As detailed in Figure 2, expert texts in the MSD dataset require a Grade 15 reading level, while the layman versions only reduce this to Grade 12.6. Similarly, SIMPWIKI's layman versions hover around Grade 11.9. Furthermore, the "Acceptability" scores indicate that while SIMPWIKI maintains high grammatical standards (>97%), the MSD dataset shows lower scores (~90%), likely due to the challenges of aligning non-parallel professional and consumer content.
The Human Evaluation Challenge
The study identified an "Expert Blind Spot" in human evaluation. Laypeople often find the "expert" source text so dense that they cannot accurately judge whether a simplification preserves the original meaning. This leads to low inter-annotator agreement (Krippendorff’s alpha of 0.299) and suggests that "simplicity" is highly subjective, contingent on the reader's prior knowledge.
Conclusion: Toward Expert-Layman Collaboration
The authors argue that the field cannot rely solely on scaling raw data to improve medical ATS. Instead, they propose a hybrid "Expert-Layman" curation framework:
- Experts ensure content preservation and identify core concepts.
- Laypeople or Medical Writers refine the language for maximum accessibility.
- Hybrid Workflows utilize automated tools (e.g., concept extraction) to bridge the knowledge gap for human annotators.
By focusing on high-quality, expert-validated parallel corpora, the medical ATS community can move toward tools that truly improve patient comprehension and health outcomes.
Summary updated