TL;DR
This study compares the performance of two advanced generative AI models, ChatGPT-4o and DeepSeek R1, in handling ophthalmological emergencies based on fictional case vignettes. While both models show promise, DeepSeek R1 generally provided more accurate and safer advice, though ChatGPT-4o's interactive questioning style more closely mimics a medical history-taking process.
What Problem Does This Paper Address?
The increasing accessibility of generative AI tools like ChatGPT raises questions about their reliability for medical advice, particularly in urgent situations. While previous research has assessed older AI models (e.g., ChatGPT-3.5) for ophthalmological emergencies, this paper addresses the need to evaluate newer, more capable models like ChatGPT-4o and DeepSeek R1. The core problem is to understand how these advanced AIs perform in diagnosing, triaging, and providing pre-hospital management advice for acute eye conditions, and whether they can be safely used by laypersons seeking immediate guidance.
Methods
The researchers employed a structured approach using ten fictional ophthalmological case vignettes. These vignettes, designed to mimic layperson descriptions of symptoms, were presented to both ChatGPT-4o and DeepSeek R1. To ensure robustness, each vignette was entered into the AI five times, resulting in 50 responses per AI. The interaction followed a predefined path: an initial prompt with the vignette and a question about diagnosis/treatment, followed by a second prompt to clarify urgency if a physician visit was recommended, or to confirm the need for consultation otherwise. A third prompt was available for further clarification at the researchers' discretion. Crucially, the study adhered to a simple prompting protocol to simulate layperson usage, avoiding advanced techniques. Responses were then meticulously analyzed using a standardized manual, evaluating metrics such as diagnostic specificity, triage accuracy, treatment accuracy, and the presence of potentially harmful information.
Datasets
The study did not use datasets for training the AI models. Instead, it utilized a set of ten custom-made fictional case vignettes that were previously published and developed by the authors. These vignettes were designed to represent a range of ophthalmological symptoms and urgency levels. The evaluation of the AI responses was based on a previously published standardized manual, which served as the benchmark for analysis.
Key Insights and Ideas
- DeepSeek R1's Edge in Content Quality: Despite ChatGPT-4o's more conversational approach, DeepSeek R1 consistently outperformed it in several critical content-related metrics, including treatment accuracy and the reflection of symptom severity.
- The Double-Edged Sword of Interaction: ChatGPT-4o's tendency to ask clarifying questions, while mimicking a doctor's history-taking, could be misleading. It might create an illusion of trustworthiness that doesn't always align with the accuracy or safety of its advice. DeepSeek R1, by contrast, was more direct and less interactive, but provided more specific and safer recommendations.
- Harmful Advice Remains a Concern: Even with these advanced models, the study found a significant percentage of potentially harmful recommendations. This underscores that neither AI should be relied upon as a sole source of information for medical emergencies.
- Disclaimer Effectiveness is Nuanced: The presence of a disclaimer (stating the AI is not a doctor) did not consistently correlate with reduced harm. For ChatGPT-4o, the absence of a disclaimer was a weak indicator of higher potential harm, suggesting a complex relationship. DeepSeek R1, however, more frequently provided disclaimers.
- No Clear Winner in Triage: While ChatGPT-4o showed a slight edge in triage accuracy (73% vs. 66%), the difference was not statistically significant, and the overall performance in this area was comparable.
Key Results
- Treatment Accuracy: DeepSeek R1 achieved 60% treatment accuracy, compared to 50% for ChatGPT-4o.
- Wrong/Conflicting Information: DeepSeek R1 had a lower share of answers with wrong (46% vs. 60%) or conflicting (30% vs. 40%) information.
- Symptom Severity Reflection: DeepSeek R1's answers correctly captured symptom severity in 98% of cases, significantly higher than ChatGPT-4o's 78%.
- Potentially Harmful Answers: DeepSeek R1 provided potentially harmful answers in 38% of cases, compared to 50% for ChatGPT-4o.
- Specific Treatment Advice: DeepSeek R1 provided specific treatment advice more frequently (42% vs. 20%).
- Diagnostic Accuracy: Both models achieved 100% diagnostic accuracy when they provided a single diagnosis.
- Triage Accuracy: ChatGPT-4o performed slightly better at 73% compared to DeepSeek R1's 66%.
- Interactive Questions: ChatGPT-4o asked questions back to the user in 100% of instances, while DeepSeek R1 did so in 0%.
Limitations and Future Work
The authors acknowledge several limitations. The use of fictional vignettes, while mimicking layperson queries, may not perfectly represent real-world patient interactions. The study's methodology was primarily designed to evaluate triage, appropriateness of pre-hospital measures, and potential for harm, rather than definitive diagnostic or treatment accuracy. Furthermore, the qualitative nature of the evaluations, conducted at different times, might reduce direct comparability with previous studies.
Future work could involve exploring the interactive capabilities of ChatGPT-4o more deeply, as continued dialogue might yield better results. The researchers also suggest that further investigation into the cost-effectiveness of these models, as hinted at in other studies, would be valuable. Ultimately, while these AI models represent significant advancements, the observed potential for harmful recommendations means they are not yet recommended as a sole source of information for ophthalmological emergencies for the general public.
Summary updated