Preliminary Evaluation of an Open-Source LLM for Lay Translation of German Clinical Documents

Tabea M. G. Pakull, Amin Dada, Hendrik Damm, Anke Fleischhauer, Sven Benson, Noëlle Bender, Nicola Prasuhn, Karol Kamiński, Christoph Friedrich, Péter Horn, Jens Kleesiek, Dirk Schadendorf, Ina Pretzell2025 · DOI

Abstract

Tabea Pakull, Amin Dada, Hendrik Damm, Anke Fleischhauer, Sven Benson, Noëlle Bender, Nicola Prasuhn, Katharina Kaminski, Christoph Friedrich, Peter Horn, Jens Kleesiek, Dirk Schadendorf, Ina Pretzell. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health).

2025.

Introduction

In modern medicine, a significant gap exists between technical clinical documentation and patient comprehension. This is particularly evident in Molecular Tumor Board (MTB) protocols, which detail personalized cancer treatments based on complex genomic data. While Large Language Models (LLMs) offer a potential solution for automated lay translation, the German healthcare system's strict adherence to GDPR makes the use of closed-source, cloud-based models problematic. This study evaluates the efficacy of an open-source alternative, Llama-3.3-70B-Instruct, hosted locally to ensure data privacy.

Methodology

The researchers compared two prompting strategies using the Llama-3.3-70B-Instruct model:

  • Zero-shot: Direct instructions to simplify technical text.
  • Few-shot: Instructions supplemented with 1–3 examples of technical text paired with expert-validated lay translations.

Evaluation was conducted across two datasets: the GRASCCO (German Synthetic Clinical Corpus) and a high-value set of real MTB Protocols from a German university hospital. Performance was measured using traditional readability indices (LIX, WSTF, FRE_de), LLM-as-a-Judge (G-Eval), similarity metrics (BERTScore, ROUGE-1), and manual error analysis.

Results and Performance Metrics

Quantitative analysis demonstrated that the LLM significantly improved the readability of clinical texts. On the GRASCCO dataset, the German Flesch Reading Ease (FRE_de) score increased from a baseline of 21.4 (highly technical) to 39.3 (improved, though still challenging).

Table 1: Performance metrics across datasets and prompting strategies.

As shown in the performance table, few-shot prompting consistently outperformed zero-shot approaches for MTB protocols, achieving higher alignment with human "gold standard" translations (BERTScore of 0.738 vs. 0.687). Notably, while G-Eval scores for comprehensibility were high, the model occasionally struggled to distinguish between necessary medical omissions and unintentional factual errors.

Error Analysis and Qualitative Insights

Manual review revealed that the most frequent challenge for the model was providing excessive detail. Zero-shot prompting often resulted in summaries that remained "Too Detailed" for a lay audience. However, the introduction of few-shot examples significantly mitigated this issue, reducing instances of excessive detail from 39 to 23.

Figure 1: Distribution of error types for Zero-Shot vs. Few-Shot prompting.

Despite these improvements, a "complexity floor" remains; MTB protocols are inherently technical, and even optimized translations require a relatively high level of literacy. Factual errors, while rare, persisted in both prompting styles, reinforcing the necessity of clinician oversight in the translation pipeline.

Conclusion

Llama-3.3-70B-Instruct proves to be a viable open-source tool for clinical lay translation, particularly when utilizing few-shot prompting to align with expert styles. While the model significantly lowers the barrier to understanding, it should be viewed as a "copilot" for medical professionals rather than a fully autonomous system. Future work will explore Retrieval-Augmented Generation (RAG) to further enhance factual accuracy and investigate the direct impact of these translations on patient well-being.

Summary updated

BibTeX
@misc{pakull2025preliminary,
  author = {Tabea M. G. Pakull and Amin Dada and Hendrik Damm and Anke Fleischhauer and Sven Benson and Noëlle Bender and Nicola Prasuhn and Karol Kamiński and Christoph Friedrich and Péter Horn and Jens Kleesiek and Dirk Schadendorf and Ina Pretzell},
  title = {{Preliminary Evaluation of an Open-Source LLM for Lay Translation of German Clinical Documents}},
  year = {2025},
  doi = {10.18653/v1/2025.cl4health-1.15},
}
Figure