Abstract
Tabea Pakull, Amin Dada, Hendrik Damm, Anke Fleischhauer, Sven Benson, Noëlle Bender, Nicola Prasuhn, Katharina Kaminski, Christoph Friedrich, Peter Horn, Jens Kleesiek, Dirk Schadendorf, Ina Pretzell. Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health).
2025.
Introduction
In modern medicine, a significant gap exists between technical clinical documentation and patient comprehension. This is particularly evident in Molecular Tumor Board (MTB) protocols, which detail personalized cancer treatments based on complex genomic data. While Large Language Models (LLMs) offer a potential solution for automated lay translation, the German healthcare system's strict adherence to GDPR makes the use of closed-source, cloud-based models problematic. This study evaluates the efficacy of an open-source alternative, Llama-3.3-70B-Instruct, hosted locally to ensure data privacy.
Methodology
The researchers compared two prompting strategies using the Llama-3.3-70B-Instruct model:
- Zero-shot: Direct instructions to simplify technical text.
- Few-shot: Instructions supplemented with 1–3 examples of technical text paired with expert-validated lay translations.
Evaluation was conducted across two datasets: the GRASCCO (German Synthetic Clinical Corpus) and a high-value set of real MTB Protocols from a German university hospital. Performance was measured using traditional readability indices (LIX, WSTF, FRE_de), LLM-as-a-Judge (G-Eval), similarity metrics (BERTScore, ROUGE-1), and manual error analysis.
Results and Performance Metrics
Quantitative analysis demonstrated that the LLM significantly improved the readability of clinical texts. On the GRASCCO dataset, the German Flesch Reading Ease (FRE_de) score increased from a baseline of 21.4 (highly technical) to 39.3 (improved, though still challenging).
As shown in the performance table, few-shot prompting consistently outperformed zero-shot approaches for MTB protocols, achieving higher alignment with human "gold standard" translations (BERTScore of 0.738 vs. 0.687). Notably, while G-Eval scores for comprehensibility were high, the model occasionally struggled to distinguish between necessary medical omissions and unintentional factual errors.
Error Analysis and Qualitative Insights
Manual review revealed that the most frequent challenge for the model was providing excessive detail. Zero-shot prompting often resulted in summaries that remained "Too Detailed" for a lay audience. However, the introduction of few-shot examples significantly mitigated this issue, reducing instances of excessive detail from 39 to 23.
Despite these improvements, a "complexity floor" remains; MTB protocols are inherently technical, and even optimized translations require a relatively high level of literacy. Factual errors, while rare, persisted in both prompting styles, reinforcing the necessity of clinician oversight in the translation pipeline.
Conclusion
Llama-3.3-70B-Instruct proves to be a viable open-source tool for clinical lay translation, particularly when utilizing few-shot prompting to align with expert styles. While the model significantly lowers the barrier to understanding, it should be viewed as a "copilot" for medical professionals rather than a fully autonomous system. Future work will explore Retrieval-Augmented Generation (RAG) to further enhance factual accuracy and investigate the direct impact of these translations on patient well-being.
Summary updated