Comparison of ChatGPT-4o and DeepSeek R1 in the Management of Ophthalmological Emergencies—An Analysis of Ten Fictional Case Vignettes

Dominik Knebel, Siegfried Priglinger, Benedikt SchwormJournal of Clinical Medicine 2025 · DOI

TL;DR

This study compares the performance of two advanced generative AI models, ChatGPT-4o and DeepSeek R1, in handling ophthalmological emergencies based on fictional case vignettes. While both models show promise, DeepSeek R1 generally provided more accurate and safer advice, though ChatGPT-4o's interactive questioning style more closely mimics a medical history-taking process.

What Problem Does This Paper Address?

The increasing accessibility of generative AI tools like ChatGPT raises questions about their reliability for medical advice, particularly in urgent situations. While previous research has assessed older AI models (e.g., ChatGPT-3.5) for ophthalmological emergencies, this paper addresses the need to evaluate newer, more capable models like ChatGPT-4o and DeepSeek R1. The core problem is to understand how these advanced AIs perform in diagnosing, triaging, and providing pre-hospital management advice for acute eye conditions, and whether they can be safely used by laypersons seeking immediate guidance.

Methods

The researchers employed a structured approach using ten fictional ophthalmological case vignettes. These vignettes, designed to mimic layperson descriptions of symptoms, were presented to both ChatGPT-4o and DeepSeek R1. To ensure robustness, each vignette was entered into the AI five times, resulting in 50 responses per AI. The interaction followed a predefined path: an initial prompt with the vignette and a question about diagnosis/treatment, followed by a second prompt to clarify urgency if a physician visit was recommended, or to confirm the need for consultation otherwise. A third prompt was available for further clarification at the researchers' discretion. Crucially, the study adhered to a simple prompting protocol to simulate layperson usage, avoiding advanced techniques. Responses were then meticulously analyzed using a standardized manual, evaluating metrics such as diagnostic specificity, triage accuracy, treatment accuracy, and the presence of potentially harmful information.

Datasets

The study did not use datasets for training the AI models. Instead, it utilized a set of ten custom-made fictional case vignettes that were previously published and developed by the authors. These vignettes were designed to represent a range of ophthalmological symptoms and urgency levels. The evaluation of the AI responses was based on a previously published standardized manual, which served as the benchmark for analysis.

Key Insights and Ideas

  • DeepSeek R1's Edge in Content Quality: Despite ChatGPT-4o's more conversational approach, DeepSeek R1 consistently outperformed it in several critical content-related metrics, including treatment accuracy and the reflection of symptom severity.
  • The Double-Edged Sword of Interaction: ChatGPT-4o's tendency to ask clarifying questions, while mimicking a doctor's history-taking, could be misleading. It might create an illusion of trustworthiness that doesn't always align with the accuracy or safety of its advice. DeepSeek R1, by contrast, was more direct and less interactive, but provided more specific and safer recommendations.
  • Harmful Advice Remains a Concern: Even with these advanced models, the study found a significant percentage of potentially harmful recommendations. This underscores that neither AI should be relied upon as a sole source of information for medical emergencies.
  • Disclaimer Effectiveness is Nuanced: The presence of a disclaimer (stating the AI is not a doctor) did not consistently correlate with reduced harm. For ChatGPT-4o, the absence of a disclaimer was a weak indicator of higher potential harm, suggesting a complex relationship. DeepSeek R1, however, more frequently provided disclaimers.
  • No Clear Winner in Triage: While ChatGPT-4o showed a slight edge in triage accuracy (73% vs. 66%), the difference was not statistically significant, and the overall performance in this area was comparable.

Key Results

  • Treatment Accuracy: DeepSeek R1 achieved 60% treatment accuracy, compared to 50% for ChatGPT-4o.
  • Wrong/Conflicting Information: DeepSeek R1 had a lower share of answers with wrong (46% vs. 60%) or conflicting (30% vs. 40%) information.
  • Symptom Severity Reflection: DeepSeek R1's answers correctly captured symptom severity in 98% of cases, significantly higher than ChatGPT-4o's 78%.
  • Potentially Harmful Answers: DeepSeek R1 provided potentially harmful answers in 38% of cases, compared to 50% for ChatGPT-4o.
  • Specific Treatment Advice: DeepSeek R1 provided specific treatment advice more frequently (42% vs. 20%).
  • Diagnostic Accuracy: Both models achieved 100% diagnostic accuracy when they provided a single diagnosis.
  • Triage Accuracy: ChatGPT-4o performed slightly better at 73% compared to DeepSeek R1's 66%.
  • Interactive Questions: ChatGPT-4o asked questions back to the user in 100% of instances, while DeepSeek R1 did so in 0%.

Limitations and Future Work

The authors acknowledge several limitations. The use of fictional vignettes, while mimicking layperson queries, may not perfectly represent real-world patient interactions. The study's methodology was primarily designed to evaluate triage, appropriateness of pre-hospital measures, and potential for harm, rather than definitive diagnostic or treatment accuracy. Furthermore, the qualitative nature of the evaluations, conducted at different times, might reduce direct comparability with previous studies.

Future work could involve exploring the interactive capabilities of ChatGPT-4o more deeply, as continued dialogue might yield better results. The researchers also suggest that further investigation into the cost-effectiveness of these models, as hinted at in other studies, would be valuable. Ultimately, while these AI models represent significant advancements, the observed potential for harmful recommendations means they are not yet recommended as a sole source of information for ophthalmological emergencies for the general public.

Summary updated

Abstract

<b>Background:</b> Generative artificial intelligence (AI) applications have gained increasing popularity in recent years and are used by an ever-increasing number of people on a day-to-day basis. While the performance of the earlier-generation generative AI ChatGPT-3.5 in the context of ophthalmologic emergencies has been previously assessed, the purpose of this study is to analyze the performance of the newer-generation generative AIs DeepSeek R1 (Hangzhou DeepSeek Artificial Intelligence Co., Ltd., Hangzhou, China) and ChatGPT-4o (OpenAI Inc., San Francisco, CA, USA) in the context of diagnosis, triage and prehospital management of ophthalmological emergencies. <b>Methods:</b> Ten previously published fictional case vignettes representing queries in the English language of patients experiencing acute ophthalmological symptoms were entered into the generative AIs DeepSeek R1 and ChatGPT-4o. The interaction with the generative AIs followed a previously described structured interaction path. In a random order, each case vignette was entered into separate chats five times, producing a total of 50 answers from each generative AI. Each answer was analyzed according to a previously published manual. <b>Results:</b> We observed better values for DeepSeek R1 compared to ChatGPT-4o in terms of treatment accuracy (60% compared to 50%), the share of answers containing wrong (46% compared to 60%) or conflicting information (30% compared to 40%), the share of answers that correctly captured the overall severity of symptoms (98% compared to 78%), as well as the share of potentially harmful answers (38% compared to 50%). Moreover, DeepSeek R1 more frequently provided a single diagnosis (20% compared to 16%) and specific treatment advice (42% compared to 20%) than ChatGPT-4o. Both generative AIs showed a diagnostic accuracy of 100%, i.e., whenever they provided a single diagnosis, this was indeed the correct diagnosis. In terms of triage accuracy, ChatGPT-4o performed slightly better than DeepSeek R1 (73% compared to 66%). In contrast to DeepSeek R1, which never directed questions back at the user, ChatGPT-4o always did. The direction of questions at the user enables dialogues with ChatGPT-4o that more closely resemble the actual taking of a patient's history. However, DeepSeek R1 seems to perform better compared to ChatGPT-4o in terms of several important content-related metrics and has been shown to be more cost-effective in other studies. <b>Conclusions:</b> Both newer-generation generative AIs constitute remarkable milestones in the development of generative artificial intelligence. However, since potentially harmful recommendations were observed with both models, we currently do not recommend their use as sole source of information on ophthalmological emergencies for laypersons.
BibTeX
@article{knebel2025comparison,
  author = {Dominik Knebel and Siegfried Priglinger and Benedikt Schworm},
  title = {{Comparison of ChatGPT-4o and DeepSeek R1 in the Management of Ophthalmological Emergencies—An Analysis of Ten Fictional Case Vignettes}},
  journal = {Journal of Clinical Medicine},
  year = {2025},
  doi = {10.3390/jcm14248927},
}
Figure