Towards AI-supported Health Communication in Plain Language: Evaluating Intralingual Machine Translation of Medical Texts

Silvana Deilen, Ekaterina Lapshinova-Koltunski, Sergio Hernández Garrido, Christiane Maaß, Julian Hörner, Vanessa Theel, Sophie ZiemerCOLING 2024

TL;DR

This paper investigates the effectiveness of AI-driven machine translation for converting complex medical texts into "Plain German," a simplified language variety aimed at improving health literacy. While AI shows promise in enhancing readability, it still falls short of human translation quality in terms of correctness and syntactic complexity, highlighting the need for human oversight.

What Problem Does This Paper Address?

Health literacy – the ability to find, understand, and use health information – is a critical determinant of health outcomes. In Germany, a significant portion of the population struggles with comprehending health-related information, leading to the development of initiatives like the National Action Plan Health Literacy. A key strategy within this plan is the use of "Plain Language" (and its more simplified variant, "Easy Language") to make complex texts more accessible.

Traditionally, this simplification has been a manual process. However, the authors explore the potential of machine translation (MT) systems, specifically for intralingual translation (translating within the same language, from standard to Plain German). This is a less explored area for MT compared to interlingual translation. The core problem is to determine if current AI tools can effectively and accurately produce Plain German medical texts, thereby supporting the goal of accessible health communication and potentially reducing the workload for human translators.

Methods

The study employed a multi-faceted evaluation approach for machine-translated medical texts into Plain German.

  • Data Collection: Thirty medical texts were selected from the German health magazine "Apotheken Umschau." For each text, both the original "source text" and a human-translated "Plain German" version were available. These texts were then machine-translated into Plain German using the SUMM AI system.
  • Correctness Evaluation: This was a manual assessment by two researchers, with consultation from a healthcare professional. They checked if the medical information remained valid and accurate after simplification. A single content-related error was enough to deem a translation incorrect, as the focus was on user safety.
  • Readability Assessment: The Hohenheim Comprehensibility Index (HIX) was used to quantitatively measure readability. HIX is a meta-index combining four major readability formulas. A benchmark of 16 points was set for Plain German texts.
  • Syntactic Complexity Analysis: Dependency parsing was used to identify and quantify specific syntactic relations (e.g., adnominal clauses, adverbial clauses). The frequency of these relations was compared across source texts, human translations, and machine translations, with a higher frequency indicating greater complexity.
  • Automatic Evaluation Measures: The SARI (System for Automatic Text Simplification Evaluation) score was computed. SARI quantifies the quality of text simplification by comparing machine-translated output against source texts and human translations, measuring added, deleted, or kept words. Higher SARI scores indicate better simplification.

Datasets

  • Source Texts: Thirty medical texts from the German health magazine "Apotheken Umschau."
  • Human Translations: Corresponding Plain German versions of the thirty source texts, created by human translators and reviewed by medical professionals.
  • Machine Translations: Plain German versions of the thirty source texts generated by the SUMM AI machine translation system.

The study did not introduce a new benchmark dataset but rather used existing medical texts and their human-translated counterparts for evaluation.

Key Insights and Ideas

  • Readability is King (for AI): Machine translation systems, like the one evaluated, can significantly improve the readability of medical texts, even surpassing human translations in this regard. This is a crucial step towards making health information more accessible.
  • Complexity is the AI's Achilles' Heel: Despite improvements in readability, AI-generated Plain German texts often retain or even introduce more complex syntactic structures than human translations. This suggests that while AI can simplify vocabulary and sentence length, it struggles with the nuanced simplification of grammatical structures.
  • Correctness is Non-Negotiable: For safety-critical domains like healthcare, even a single content error in a machine translation is unacceptable for direct end-user consumption. The study found that the evaluated AI system produced a high number of errors, necessitating extensive post-editing.
  • Human Touch Remains Essential: The findings strongly suggest that AI-generated Plain German texts are not yet ready for direct use by the public. They are best viewed as a tool to assist professional translators and experts, reducing their workload rather than replacing them.

Key Results

  • Readability: Machine translations achieved the highest comprehensibility (mean HIX of 19.15), followed by human translations (mean HIX of 17.74), with source texts being the least comprehensible (mean HIX of 10.46). All machine-translated texts met the benchmark for Plain German.
  • Correctness: Only one out of thirty machine-translated texts was deemed fully correct. The other 29 contained various errors, including grammatical mistakes, misspellings, omissions, incorrect explanations of technical terms, and semantic inaccuracies.
  • Syntactic Complexity: Both source texts and machine translations exhibited a higher number of complex syntactic relations than human translations. Notably, machine translations were often more complex than the source texts themselves.
  • SARI Score: The system achieved an average SARI score of 40.67, which is in line with state-of-the-art text simplification models. However, the presence of many outliers and segments scoring over 55 suggests inconsistency.

Limitations and Future Work

  • Limitations:
    • The study focused solely on the text perspective, not the user perspective.
  • The correctness evaluation was qualitative and did not involve a quantitative error analysis or ranking of error severity.
  • The analysis of syntactic complexity was based on specific dependency relations and might not capture all aspects of complexity.
  • Future Work:
    • Conduct empirical studies involving eye-tracking and reading experiments to understand cognitive processing costs for target users.
  • Use questionnaires to assess end-user acceptance of generated texts.
  • Classify and rank different types of machine translation errors.
  • Investigate cases of partial correctness where only some information is incorrect or missing.
  • Evaluate improved versions of the SUMM AI system to track progress.
  • Explore the role of professional post-editing skills in conjunction with AI tools.

Summary updated

BibTeX
@inproceedings{deilen2024towards,
  author = {Silvana Deilen and Ekaterina Lapshinova-Koltunski and Sergio Hernández Garrido and Christiane Maaß and Julian Hörner and Vanessa Theel and Sophie Ziemer},
  title = {{Towards AI-supported Health Communication in Plain Language: Evaluating Intralingual Machine Translation of Medical Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure