Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment

Chenxu Wang, Shuhan Li, Nuoxi Lin, Xinyu Zhang, Ying Han, Xiandi Wang, Di Liu, Xiaomei Tan, Dan Pu, Kang Li, Guangwu Qian, Rong YinJournal of Medical Internet Research 2024 · DOI

TL;DR

This paper explores the potential of using ChatGPT as a cost-effective, simulated standardized patient for medical training evaluations. By employing prompt engineering, the researchers significantly improved ChatGPT's accuracy, realism, and adaptability, demonstrating its viability for history-taking tasks and its potential to enhance medical education.

What Problem Does This Paper Address?

Traditional medical training and assessment often rely on standardized patients (SPs) – individuals trained to simulate specific patient conditions. While effective, this method is labor-intensive, costly, and can be inaccessible, particularly in resource-limited settings. The paper addresses the need for more accessible and scalable alternatives by investigating whether large language models (LLMs) like ChatGPT can effectively fulfill the role of an SP, specifically for history-taking simulations. The challenge lies in ensuring the LLM's responses are clinically accurate, realistic, and consistent, mimicking the nuances of human interaction in a medical context.

Methods

The study employed a two-phase experimental approach:

  • Phase 1: Feasibility Test: ChatGPT was tasked with simulating conversations for inflammatory bowel disease (IBD) scenarios across three quality groups: poor, medium, and good inquiries. Responses were evaluated for relevance and accuracy.
  • Phase 2: Performance Enhancement and Prompt Engineering: This phase focused on refining ChatGPT's performance. The researchers developed and tested revised prompts (RP) based on identified shortcomings in the initial prompts (OP). Key evaluation criteria included anthropomorphism (realism), clinical accuracy, and adaptability. A comparative analysis was conducted between OP and RP performance, with a total of 300 runs. The generalizability of the revised prompt was further tested with a different script for 60 runs.

Datasets

The study did not explicitly mention pre-existing datasets for training ChatGPT. Instead, it utilized a simulated medical record for Inflammatory Bowel Disease (IBD) as the basis for the standardized patient scenarios. The "datasets" in this context refer to the interactions generated between the researchers (acting as medical students) and ChatGPT, categorized by inquiry quality (poor, medium, good) and prompt type (original vs. revised).

Key Insights and Ideas

  • LLMs as a Viable Alternative: ChatGPT, with appropriate prompting, can effectively simulate standardized patients for medical history-taking, offering a cost-effective and scalable solution.
  • The Power of Prompt Engineering: The study strongly emphasizes that prompt engineering is crucial for optimizing LLM performance in specialized tasks like medical simulation. Simple adjustments to prompts can dramatically improve accuracy, realism, and adaptability.
  • Anthropomorphism is Key: For realistic medical simulation, ChatGPT needs to go beyond just providing factual information. It must exhibit human-like qualities, including appropriate tone, avoiding overly technical jargon, and even simulating emotional responses like anxiety when prompted.
  • Language Independence: The study found that the language used for prompts and dialogues (English vs. Chinese) had a nonsignificant impact on ChatGPT's performance, suggesting its core capabilities are language-agnostic in this context.

Key Results

  • Feasibility: ChatGPT demonstrated the ability to differentiate between poor, medium, and good medical inquiries with statistically significant score differences between the poor and medium/good groups.
  • Performance Enhancement: Revised prompts (RP) significantly improved ChatGPT's performance. The score difference percentage (SDP), a measure of error rate, dropped from 29.83% with original prompts (OP) to 6.06% with RP. The standard deviation (SD) also decreased substantially.
  • Accuracy Improvement: ChatGPT's score accuracy improved by approximately 4.926 times compared to unrevised prompts.
  • Generalizability: The revised prompts maintained their effectiveness when tested on a different medical script, indicating good generalizability.
  • Language Impact: The influence of language (English vs. Chinese) on chatbot performance was found to be insignificant.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that ChatGPT's knowledge base is not continuously updated, which could affect its performance with newer diseases or medical information.
  • The evaluation criteria for ChatGPT were primarily objective, lacking the subjective grading that is often part of real standardized patient assessments.
  • While improved, ChatGPT's responses can still exhibit randomness, and it may not always initiate subjective emotional responses without explicit prompting.
  • The study used simplified Chinese for dialogues but English for prompts, and while language impact was found to be insignificant, further exploration with various language combinations is suggested.
  • Future Work:
    • Future studies should explore more comprehensive prompt engineering to further reduce the SDP and enhance accuracy.
  • Investigating the impact of subjective grading and more nuanced emotional simulation in prompts is recommended.
  • Testing ChatGPT's performance with other languages and corresponding prompt engineering is suggested.
  • Further research is needed to fully integrate ChatGPT into standardized patient training and assess its long-term impact on medical education.

Summary updated

Abstract

ChatGPT, as a representative LLM, is a viable tool for simulating standardized patients in medical assessments, with the potential to enhance medical training. By incorporating proper prompts, ChatGPT's scoring accuracy and response realism significantly improved, approaching the feasibility of actual clinical use. Also, the influence of the adopted language is nonsignificant on the outcome of the chatbot.
BibTeX
@article{wang2024application,
  author = {Chenxu Wang and Shuhan Li and Nuoxi Lin and Xinyu Zhang and Ying Han and Xiandi Wang and Di Liu and Xiaomei Tan and Dan Pu and Kang Li and Guangwu Qian and Rong Yin},
  title = {{Application of Large Language Models in Medical Training Evaluation—Using ChatGPT as a Standardized Patient: Multimetric Assessment}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/59435},
}
Figure