TL;DR
This paper investigates the safety and appropriateness of ChatGPT's responses to real-world patient questions following cataract surgery. While LLMs show promise in providing helpful answers, the study highlights significant limitations regarding potential harm and adherence to clinical consensus, underscoring the need for caution in healthcare applications.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has sparked interest in their application across various fields, including healthcare. However, there's a critical lack of data on their safety and performance in real-world clinical scenarios. Healthcare interactions are inherently conversational, making LLMs a natural fit, but their complexities around reliability and safety are significant hurdles. While efforts like fine-tuning and prompt engineering exist, a lack of standardized evaluation metrics for healthcare makes assessing LLM performance challenging. This paper specifically addresses this gap by evaluating how a popular LLM, ChatGPT, handles patient inquiries post-cataract surgery, a common procedure with significant implications for patient well-being.
Methods
The researchers employed a two-pronged approach to evaluate ChatGPT's responses. First, they collected a dataset of 131 unique questions from 120 patients who had undergone routine cataract surgery. These questions were gathered via an autonomous telemedicine clinical assistant designed to elicit post-operative concerns.
For model inference, they used a single zero-shot text prompt. This prompt was iteratively refined with clinician input to ensure it was conversational, provided context (including optional symptom information), and instructed the model to act as a helpful, scientifically grounded assistant. Two variations of the prompt were used: one with patient symptom context and one without.
The evaluation of ChatGPT's responses was conducted by two ophthalmologists who independently assessed the answers based on a seven-axis framework adapted from the Med-PaLM evaluation. These axes covered:
- Intent and Helpfulness: Does it address the question's intent? How helpful is the answer?
- Clinical Harm: What is the likelihood and extent of possible harm?
- Clinical Appropriateness: Is the answer aligned with clinical/scientific consensus? Is there inappropriate or incorrect content? Is there evidence of clinical reasoning?
Datasets
The primary dataset used in this study consists of 131 unique patient questions collected from 120 patients following routine cataract surgery. These questions were gathered through an autonomous telemedicine clinical assistant. The researchers did not introduce a new benchmark dataset but rather adapted existing evaluation strategies from prior work (specifically, the Med-PaLM evaluation framework).
Key Insights and Ideas
- LLMs can be helpful for routine queries: The study demonstrates that LLMs like ChatGPT have the potential to provide helpful answers to common patient questions after surgery, offering a scalable solution for post-operative care.
- Safety is a major concern: Despite helpfulness, the potential for harm, including "moderate or mild harm" and even "sight loss or severe harm" in some instances, is a critical limitation. This highlights the need for robust safety mechanisms.
- Clinical consensus is not always met: A significant portion of responses were found to be opposed to or lacking clear alignment with clinical or scientific consensus, raising concerns about the accuracy and reliability of LLM-generated medical advice.
- Symptom context can be a double-edged sword: Adding symptom information to the prompt did not improve the likelihood of clinical reasoning and, in fact, increased the proportion of answers with inappropriate or incorrect content. This suggests that current LLMs may struggle to effectively integrate complex clinical context.
- Question clarity matters: A notable percentage of questions were not clearly understood, often due to mistranscriptions from the voice-based assistant or because patients posed statements rather than direct questions.
Key Results
- On average, most answers (71.0%) were rated as addressing the question's intent.
- 59.9% of responses were rated 'helpful', with an additional 36.3% rated 'somewhat helpful'.
- 92.7% of responses were rated as 'low' likelihood of harm.
- However, 24.4% of answers had the possibility of 'moderate or mild harm', and a few responses carried the risk of "sight loss or severe harm."
- 9.5% of answers were opposed to clinical or scientific consensus.
- 53.4% of answers contained inappropriate or incorrect content when symptom information was provided.
- 51.9% of responses showed evidence of clinical reasoning when symptom information was provided, a significant increase from 2.3% without symptom information. However, this did not correlate with improved safety or appropriateness.
- 71.0% of responses had "no opportunity for reasoning" when symptom information was not provided.
Limitations and Future Work
The authors acknowledge several limitations:
- Minimal prompt engineering: The use of a simple zero-shot prompt could be improved with more sophisticated techniques like few-shot or chain-of-thought examples, which have been shown to enhance accuracy.
- Lack of human expert comparison: The study did not directly compare LLM responses to those of human experts, which is crucial for a comprehensive assessment of appropriateness and safety.
- Potential for speech-to-text errors: The use of a voice-based assistant introduced mistranscriptions, impacting question clarity and potentially response quality.
Future work could explore:
- Advanced prompting strategies: Investigating few-shot learning, chain-of-thought prompting, self-consistency decoding, and retrieval augmentation to improve LLM performance and reliability in healthcare.
- Direct comparison with human experts: Establishing a benchmark against human clinician responses to better gauge LLM capabilities and limitations.
- Developing specialized healthcare LLMs: Further research into fine-tuning LLMs specifically for medical domains to enhance accuracy and safety.
Summary updated