A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports

Daniel Truhn, Christian David Weber, Benedikt J. Braun, Keno K. Bressem, Jakob Nikolas Kather, Christiane Kühl, Sven NebelungScientific 2023 · DOI

Abstract

Large language models (LLMs) have shown potential in various applications, including clinical practice. However, their accuracy and utility in providing treatment recommendations for orthopedic conditions remain to be investigated. Thus, this pilot study aims to evaluate the validity of treatment recommendations generated by GPT-4 for common knee and shoulder orthopedic conditions using anonymized clinical MRI reports. A retrospective analysis was conducted using 20 anonymized clinical MRI reports, with varying severity and complexity. Treatment recommendations were elicited from GPT-4 and evaluated by two board-certified specialty-trained senior orthopedic surgeons. Their evaluation focused on semiquantitative gradings of accuracy and clinical utility and potential limitations of the LLM-generated recommendations. GPT-4 provided treatment recommendations for 20 patients (mean age, 50 years ± 19; 12 men) with acute and chronic knee and shoulder conditions. The LLM produced largely accurate and clinically useful recommendations. However, limited awareness of a patient's overall situation, a tendency to incorrectly appreciate treatment urgency, and largely schematic and unspecific treatment recommendations were observed and may reduce its clinical usefulness. In conclusion, LLM-based treatment recommendations are largely adequate and not prone to 'hallucinations', yet inadequate in particular situations. Critical guidance by healthcare professionals is obligatory, and independent use by patients is discouraged, given the dependency on precise data input.

Introduction

As Large Language Models (LLMs) like GPT-4 become ubiquitous, patients are increasingly turning to AI for medical advice, and clinicians are exploring their use as "co-pilots." However, the stakes in orthopedics are high—misinterpreting an MRI report can lead to delayed surgery or inappropriate rehabilitation. This study investigates whether GPT-4 can translate complex, technical MRI findings into valid, prioritized treatment recommendations without falling victim to "hallucinations" (confident but false claims).

Methods

The researchers conducted a retrospective analysis using a dataset of 20 anonymized clinical MRI reports (10 knee, 10 shoulder) from a tertiary academic medical center in Germany. The cases ranged from routine degenerative changes to complex multi-ligament injuries and acute infections.

The study employed a three-step standardized prompting sequence to generate recommendations, which were then evaluated by two board-certified senior orthopedic surgeons using 5-point Likert scales.

Workflow of the AI-based treatment recommendation process, from clinical routine to surgeon evaluation.

Key Results

The evaluation revealed that GPT-4's recommendations were largely accurate and scientifically sound. Notably, the study found zero instances of "hallucinations," likely due to the high-quality, structured data provided in the professional MRI reports.

Surgeon evaluations of GPT-4 recommendations across five categories for knee and shoulder cases.

Key findings include:

  • Overall Quality: Rated as "good" or better for both knee and shoulder cases.
  • Consistency: 100% of recommendations were considered "up-to-date" and "consistent."
  • Anatomical Performance: Recommendations for shoulder cases generally received higher ratings than those for knee cases.

Discussion and Clinical Insights

While the results are promising, several critical limitations were identified. GPT-4 exhibited a "schematic bias," often defaulting to a rigid hierarchy of conservative treatments (Rest, Ice, Physical Therapy) before considering surgery. This led to "epidemiologic dominance" errors, such as recommending weight loss for shoulder issues—a standard protocol for knee osteoarthritis but less relevant for the shoulder.

Most concerning was the AI’s failure to recognize medical urgency. In a case of septic arthritis, GPT-4 failed to emphasize the need for immediate surgical intervention. It also missed the risk of popliteal artery injury in a knee dislocation, a life-threatening complication that occurs in approximately 10% of such cases.

Conclusion

GPT-4 is a powerful tool for translating technical medical data into structured advice, but it lacks the nuanced judgment required for emergency triage. While it can serve as a valuable assistant for professionals, independent use by patients is strongly discouraged. Future work should explore multimodal models that can analyze MRI images directly and fine-tune LLMs on domain-specific medical datasets to provide more tailored, patient-specific recommendations.

Summary updated

Abstract

Large language models (LLMs) have shown potential in various applications, including clinical practice. However, their accuracy and utility in providing treatment recommendations for orthopedic conditions remain to be investigated. Thus, this pilot study aims to evaluate the validity of treatment recommendations generated by GPT-4 for common knee and shoulder orthopedic conditions using anonymized clinical MRI reports. A retrospective analysis was conducted using 20 anonymized clinical MRI reports, with varying severity and complexity. Treatment recommendations were elicited from GPT-4 and evaluated by two board-certified specialty-trained senior orthopedic surgeons. Their evaluation focused on semiquantitative gradings of accuracy and clinical utility and potential limitations of the LLM-generated recommendations. GPT-4 provided treatment recommendations for 20 patients (mean age, 50 years ± 19 [standard deviation]; 12 men) with acute and chronic knee and shoulder conditions. The LLM produced largely accurate and clinically useful recommendations. However, limited awareness of a patient's overall situation, a tendency to incorrectly appreciate treatment urgency, and largely schematic and unspecific treatment recommendations were observed and may reduce its clinical usefulness. In conclusion, LLM-based treatment recommendations are largely adequate and not prone to 'hallucinations', yet inadequate in particular situations. Critical guidance by healthcare professionals is obligatory, and independent use by patients is discouraged, given the dependency on precise data input.
BibTeX
@inproceedings{truhn2023pilot,
  author = {Daniel Truhn and Christian David Weber and Benedikt J. Braun and Keno K. Bressem and Jakob Nikolas Kather and Christiane Kühl and Sven Nebelung},
  title = {{A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-47500-2},
}
Figure