Comparative analysis of large language models on rare disease identification

Guangyu Ao, Min Chen, Jing Li, H. Nie, Lei Zhang, Zejun ChenOrphanet 2025 · DOI · OpenAlex

Abstract

Diagnosing rare diseases is challenging due to their low prevalence, diverse presentations, and limited recognition, often leading to diagnostic delays and errors. This study evaluates the effectiveness of multiple large language models (LLMs) in identifying rare diseases, comparing their performance with that of human physicians using real clinical cases. We analyzed 152 rare disease cases from the Chinese Medical Case Repository using four LLMs: ChatGPT-4o, Claude 3.5 Sonnet, Gemini Advanced, and Llama 3.1 405B. Overall, the LLMs performed better than human physicians, and Claude 3.5 Sonnet achieved the highest accuracy at 78.9%, significantly surpassing the accuracy of human physicians, which was 26.3%. These findings suggest that LLMs can improve rare disease diagnosis and serve as valuable tools in clinical settings, particularly in regions with limited resources. However, further validation and careful consideration of ethical and privacy issues are necessary for their effective integration into medical practice.

Introduction

Rare diseases represent a significant global health challenge, affecting an estimated 350 million people worldwide. By definition, these conditions are uncommon, meaning most frontline clinicians lack the specialized experience required for timely recognition. This often results in a "diagnostic odyssey," where patients endure years of misdiagnoses and unnecessary medical interventions before reaching a correct conclusion.

This study investigates whether the extensive medical knowledge encoded within Large Language Models (LLMs) can mitigate these diagnostic delays. By simulating the initial clinical encounter—using history, physical examinations, and basic laboratory data while excluding definitive genetic or biopsy results—we assess the potential of LLMs to serve as diagnostic aids in complex clinical scenarios.

Methodology

The study utilized 152 rare disease cases sourced from the Chinese Medical Case Repository (CMCR). To ensure a rigorous evaluation, cases were cross-referenced with the NIH Genetic and Rare Diseases Information Center (GARD) and the Chinese Rare Diseases List (CRDL).

Four state-of-the-art LLMs were evaluated:

  1. ChatGPT-4o (OpenAI)
  2. Claude 3.5 Sonnet (Anthropic)
  3. Gemini Advanced (Google)
  4. Llama 3.1 405B (Meta)

Each model provided a ranked list of the top five most likely diagnoses for each case. Performance was measured using two primary metrics:

  • Top-5 Accuracy: The frequency with which the correct diagnosis appeared within the model's top five suggestions.
  • Weighted Accuracy: A scoring system (1–5) reflecting the rank of the correct diagnosis (e.g., 5 points for Rank 1, 1 point for Rank 5).

A human baseline was established by recruiting three chief physicians specializing in nephrology. Notably, the complexity of the task led two physicians to withdraw; results are compared against the single physician who completed all 152 cases.

Results

The comparative analysis revealed a substantial performance gap between AI models and human expertise. Claude 3.5 Sonnet emerged as the top performer, achieving a diagnostic accuracy of 78.9%, nearly triple the 26.3% accuracy rate of the human physician.

Distribution of diagnostic ranks across the four evaluated LLMs, showing the percentage of cases where the correct diagnosis appeared in Ranks 1 through 5 versus incorrect outcomes.

Claude 3.5 Sonnet correctly identified the disease as its primary choice (Rank 1) in 64.47% of cases. In contrast, the open-source Llama 3.1 405B achieved a Rank 1 accuracy of 36.84%. The weighted accuracy scores further highlight the consistency of the models in prioritizing the correct diagnosis.

Weighted accuracy scores for the four LLMs, illustrating the relative performance based on the ranking of the correct diagnosis.

Discussion

The results underscore the "experience gap" in rare disease diagnosis. While a human physician's expertise is bound by their individual clinical history, LLMs synthesize patterns from a vast corpus of medical literature. Claude 3.5 Sonnet’s superior performance suggests a high level of context-aware reasoning, enabling it to effectively weigh disparate clinical symptoms.

The difficulty experienced by the human participants—with a 66% withdrawal rate—highlights the immense cognitive load associated with rare disease identification. LLMs could potentially serve as a "triage" tool, prompting clinicians to consider specialized testing earlier in the diagnostic process.

Limitations and Future Work

Several limitations must be acknowledged. The reliance on a Chinese case repository may introduce geographic or reporting biases. Furthermore, the study utilized static text files, whereas real-world diagnosis is an iterative, multi-turn process. The small human sample size also limits the generalizability of the baseline.

Future research should explore multi-turn diagnostic interactions and address deployment hurdles, including data privacy (HIPAA/GDPR compliance), the risk of model hallucinations, and the ethical implications of AI-assisted diagnosis. As LLMs continue to evolve, continuous re-evaluation will be necessary to maintain diagnostic standards.

Summary updated

Abstract

Diagnosing rare diseases is challenging due to their low prevalence, diverse presentations, and limited recognition, often leading to diagnostic delays and errors. This study evaluates the effectiveness of multiple large language models (LLMs) in identifying rare diseases, comparing their performance with that of human physicians using real clinical cases. We analyzed 152 rare disease cases from the Chinese Medical Case Repository using four LLMs: ChatGPT-4o, Claude 3.5 Sonnet, Gemini Advanced, and Llama 3.1 405B. Overall, the LLMs performed better than human physicians, and Claude 3.5 Sonnet achieved the highest accuracy at 78.9%, significantly surpassing the accuracy of human physicians, which was 26.3%. These findings suggest that LLMs can improve rare disease diagnosis and serve as valuable tools in clinical settings, particularly in regions with limited resources. However, further validation and careful consideration of ethical and privacy issues are necessary for their effective integration into medical practice.
BibTeX
@inproceedings{ao2025comparative,
  author = {Guangyu Ao and Min Chen and Jing Li and H. Nie and Lei Zhang and Zejun Chen},
  title = {{Comparative analysis of large language models on rare disease identification}},
  booktitle = {Orphanet},
  year = {2025},
  doi = {10.1186/s13023-025-03656-w},
}
Figure