Clinical applications of large language models in knee osteoarthritis: a systematic review

Zebing Ma, Yibing Liu, Ziyan Zhang, Suming Chen, Huayu Fan, Xiangyang Cao, Lili NiFrontiers in Medicine 2025 · DOI

TL;DR

This systematic review explores the burgeoning field of Large Language Models (LLMs) in knee osteoarthritis (KOA) management. It reveals that while LLMs show significant promise, particularly ChatGPT-4, their clinical application is still in its early stages, facing challenges in standardization, validation, and ethical considerations.

What Problem Does This Paper Address?

Knee osteoarthritis (KOA) is a widespread, debilitating condition impacting millions globally. Current diagnostic and treatment approaches often suffer from subjectivity and inefficiency. The rapid advancement of AI, especially LLMs, offers potential solutions by assisting with medical information extraction, clinical decision-making, and patient education. However, a comprehensive understanding of how LLMs are currently being applied in KOA, their effectiveness, and their methodological rigor is lacking. This paper aims to fill that gap by systematically reviewing existing research.

Methods

The authors conducted a systematic review following PRISMA guidelines, searching major databases (PubMed, Cochrane Library, Embase, Web of Science) for studies published up to June 2025. They screened studies based on predefined inclusion and exclusion criteria. Methodological quality was assessed using established tools like Cochrane RoB, STROBE, and DISCERN. Crucially, they also developed and applied two novel frameworks: CLEAR-LLM for evaluating LLM-based research methodology and CliMA-10 for assessing the quality and performance of LLM outputs in clinical applications. CliMA-10 is a multidimensional tool that considers aspects like medical accuracy, coherence, explainability, and hallucination control, with customizable weighting for different tasks.

Datasets

The paper does not introduce new datasets. Instead, it analyzes existing studies that utilized various forms of data, including:

  • Real or standardized patient cases
  • Radiological data (MRI, X-ray)
  • Clinical queries and medical records

Key Insights and Ideas

  • LLMs are not a monolith: Performance varies significantly across different LLM models (e.g., ChatGPT-4 vs. ChatGPT-3.5 vs. Gemini) and specific tasks.
  • Task matters: LLMs excel in tasks with lower clinical risk, like patient education and information summarization, but face greater challenges in high-stakes applications like diagnostic stratification and treatment planning.
  • ChatGPT-4 leads the pack: The ChatGPT-4 series, especially fine-tuned versions like Knee-ChatGPT, consistently demonstrated superior performance across various KOA-related tasks, particularly in structured output generation and clinical terminology interpretation.
  • Structured output is a key differentiator: The ability of LLMs to generate structured outputs (e.g., medical summaries, diagnostic grades) proved to be a more discriminative measure of their capabilities than open-ended tasks.
  • Hallucination remains a critical concern: Despite advancements, LLMs still risk generating factually inaccurate information, posing a significant challenge for clinical deployment.
  • Prompt engineering is crucial: Model performance can be highly sensitive to prompt design, highlighting the need for standardized and optimized prompting strategies.

Key Results

  • 16 studies were included in the review, covering various LLMs.
  • ChatGPT (various versions) was the most frequently used LLM, followed by Gemini and Claude.
  • Overall methodological quality ranged from moderate to high, but significant limitations were identified, particularly in sample representativeness and bias control.
  • Based on the CliMA-10 scores, LLMs achieved upper-moderate performance in KOA-related tasks, with an average score distribution showing a right skew.
  • The ChatGPT-4 series consistently outperformed other models, especially in structured output generation and accuracy.
  • Key limitations identified include insufficient sample representativeness, inconsistent control over hallucinations, and the lack of standardized evaluation tools.

Limitations and Future Work

The authors acknowledge several limitations:

  • Predominance of observational studies: The evidence base is largely composed of observational studies, with only two prospective interventional trials. This limits the overall evidence level.
  • Lack of long-term outcome evaluation: Most studies focus on immediate outputs and lack systematic evaluation of long-term clinical outcomes like adherence, functional recovery, or quality of life.
  • Reliance on subjective metrics: Many performance metrics depend on expert subjective ratings and Likert scales, introducing potential bias.
  • Absence of validated LLM evaluation tools: While CLEAR-LLM and CliMA-10 were developed, they are still in early stages and require further validation.
  • Limited real-world data integration: Studies often use simulated data or standardized cases, with limited integration of real-world clinical data.
  • Stochasticity of LLM outputs: The inherent randomness of LLM outputs is an overlooked issue, impacting reproducibility.

Future research directions suggested include:

  • Dialogue-based clinical management: Developing LLMs that can simulate the full patient journey through continuous, context-aware dialogue.
  • Multi-task prompt template libraries: Creating standardized, customizable prompt templates to streamline LLM application development.
  • Personalized KOA management: Enhancing LLM adaptability to real-world variables like health economics and linguistic diversity.
  • Trustworthy ILM-based decision-making: Developing evidence-tracing mechanisms and addressing output consistency to build trust.
  • Systematic cross-model comparisons: Moving beyond single-vendor reliance to conduct comprehensive comparisons of different LLMs.
  • Ethical, technical, and regulatory standardization: Establishing unified frameworks for quality assessment, ethical guidelines, and technical implementation.

Summary updated

Abstract

https://osf.io/jy4kz, identifier 10.17605/OSF.IO/479R8.
BibTeX
@inproceedings{ma2025clinical,
  author = {Zebing Ma and Yibing Liu and Ziyan Zhang and Suming Chen and Huayu Fan and Xiangyang Cao and Lili Ni},
  title = {{Clinical applications of large language models in knee osteoarthritis: a systematic review}},
  booktitle = {Frontiers in Medicine},
  year = {2025},
  doi = {10.3389/fmed.2025.1670824},
}
Figure