Language models in digital psychiatry: challenges with simplification of healthcare materials

Ankit Aich, Tingting Liu, Salvatore Giorgi, Kelsey Isman, Ruhshana Bobojonova, Lyle Ungar, Brenda CurtisNPP—Digital 2025 · DOI

Abstract: Linguistic hurdles in healthcare, such as complex language, significantly affect patient outcomes, including satisfaction with interaction, comprehension of healthcare materials, and engagement with the healthcare system. Reducing these hurdles has been a focus in healthcare delivery, as they significantly hinder patient engagement and adherence to treatments. The growing use of large language models (LLMs) in healthcare opens the possibility to reduce these linguistic hurdles. This study evaluates the ability of five prominent LLMs—GPT-3.5, GPT-4, GPT-4o, LLaMA-3, and Mistral—to simplify healthcare information to the standard recommended by the American Journal of Medicine. Our results indicate that while LLMs can approximate targeted reading levels, their outputs are inconsistent, with significant variability in reading level and deviation from the topic, making them unsuitable for deployment in healthcare settings.

Introduction

In healthcare, and specifically in psychiatry, clear communication is a clinical necessity. High "linguistic hurdles"—characterized by medical jargon and complex syntax—lead to lower patient satisfaction and poor treatment adherence. The American Journal of Medicine recommends that patient-facing materials be written at a 6th-grade reading level. However, most current resources from the CDC, WHO, and academic journals sit at a college or graduate reading level (Grade 15+).

This study investigates whether state-of-the-art large language models (LLMs) can bridge this gap by reliably "translating" high-level medical text into simplified versions. The researchers utilized a curated dataset of 25 representative documents across five critical public health topics: ADHD, Vaccinations and Immunizations, Influenza (Flu), Substance Use, and HIV. Sourced from authoritative bodies like the CDC and Mayo Clinic, the baseline Flesch-Kincaid (FK) scores for these original texts ranged from 11.2 to 27.7, highlighting the massive gap between professional medical writing and the recommended 6th-grade level.

Methodology

The study evaluated five models: GPT-3.5, GPT-4, GPT-4o, LLaMA-3, and Mistral-7B. To optimize performance, the researchers first conducted a "Prompt Selection" phase using GPT-4, testing 0-shot, 1-shot, and 2-shot paradigms. Surprisingly, 0-shot prompting—direct instruction to simplify to a specific grade level without examples—performed better than few-shot prompting for this specific task. Once the optimal prompt was identified, it was applied across all models. Readability was quantified using the Flesch-Kincaid (FK) grade level metric, which calculates complexity based on sentence length and syllable counts.

Results and Key Insights

The evaluation revealed significant challenges in using LLMs for medical simplification. While GPT-4o emerged as the most capable of reaching lower reading levels (achieving a minimum mean of 3.2 in some tests), it remained inconsistent across different topics.

Flesch-Kincaid grade levels across models for ADHD, HIV, and Substance Use topics. Note the extreme variance in LLaMA-3 for ADHD.

A critical finding was the lack of reliability and high variance in output. As shown in the ADHD data in Figure 1, LLaMA-3 exhibited extreme fluctuations, showing a maximum standard deviation of $\sigma = 27.6$. In some instances, smaller models like LLaMA-3 "cheated" the FK metric by producing long strings of punctuation and symbols instead of English text. Because the FK formula relies on word and sentence counts, this gibberish resulted in an artificially low grade level score, while other outputs were excessively complex.

Flesch-Kincaid grade levels for the Flu topic, illustrating that models often fail to reach the target 6th-grade level (dashed line).

Furthermore, models struggled significantly more when the source material was exceptionally complex. If the baseline text was at a "post-graduate" level, the models were less likely to reach the 6th-grade target successfully. For topics like HIV and Flu (Figure 2), the models often complicated the information rather than simplifying it, or suffered from "topical drift," introducing unrelated information or failing to distill core medical facts.

Discussion and Conclusion

The study highlights that while LLMs are promising, they are not yet ready for autonomous deployment in healthcare settings. The "Few-Shot Failure" observed—where adding examples actually increased text complexity—suggests that current models may over-index on the style of examples rather than the instruction to simplify.

Limitations of this study include the small sample size of 25 articles and the reliance on the Flesch-Kincaid scale, which is a mathematical proxy that does not account for conceptual difficulty or empathetic tone. Future work should explore domain-specific fine-tuning on medical datasets to improve consistency. Until LLMs become more stable, they should be viewed as tools for human editors rather than autonomous simplifiers in clinical environments.

Summary updated

Abstract

Linguistic hurdles in healthcare, such as complex language, significantly affect patient outcomes, including satisfaction with interaction, comprehension of healthcare materials, and engagement with the healthcare system. Reducing these hurdles has been a focus in healthcare delivery, as they significantly hinder patient engagement and adherence to treatments. The growing use of large language models (LLMs) in healthcare opens the possibility to reduce these linguistic hurdles. This study evaluates the ability of five prominent LLMs-GPT-3.5, GPT-4, GPT-4o, LLaMA-3, and Mistral-to simplify healthcare information to the standard recommended by the American Journal of Medicine. Our results indicate that while LLMs can approximate targeted reading levels, their outputs are inconsistent, with significant variability in reading level and deviation from the topic, making them unsuitable for deployment in healthcare settings.
BibTeX
@inproceedings{aich2025language,
  author = {Ankit Aich and Tingting Liu and Salvatore Giorgi and Kelsey Isman and Ruhshana Bobojonova and Lyle Ungar and Brenda Curtis},
  title = {{Language models in digital psychiatry: challenges with simplification of healthcare materials}},
  booktitle = {NPP—Digital},
  year = {2025},
  doi = {10.1038/s44277-025-00029-w},
}
Figure