Evaluation of the accuracy of large language models in answering bone cancer-related questions

Qi Pan, Liping Huang, Ning Liu, Feng Lin, S. YeFrontiers in Public Health 2025 · DOI

TL;DR

This study evaluates the accuracy of three leading Large Language Models (LLMs) – Deepseek V3.1, ChatGPT 5, and Grok 4 – in answering bone cancer-related questions based on NCCN clinical guidelines. All models demonstrated high accuracy, exceeding 90%, with no significant correlation found between response quality, word count, or response time.

What Problem Does This Paper Address?

Large Language Models (LLMs) are rapidly emerging as powerful tools in healthcare, offering potential for medical education, clinical decision-making, and patient care. However, their accuracy and reliability in specialized medical domains, particularly for complex diseases like bone cancer, remain a critical area of investigation. Bone cancer, encompassing diverse tumor types with intricate diagnostic criteria and treatment pathways, presents a significant challenge for LLMs. This paper aims to bridge this gap by systematically evaluating the performance of prominent LLMs in answering bone cancer-specific questions, benchmarked against established clinical guidelines.

Methods

The researchers developed a dataset of 52 questions related to bone cancer, derived from the NCCN Bone Cancer Clinical Practice Guidelines (Version 2.2025). These questions covered overview, Ewing sarcoma, and osteosarcoma domains. Each of the three LLMs (Deepseek V3.1, ChatGPT 5, and Grok 4) was queried twice within a one-month period. The generated responses were then independently evaluated and scored by two bone cancer-treating specialists using a predefined scoring rubric: Comprehensive (4 points), Correct but incomplete (3 points), Mixed with correct and incorrect/outdated data (2 points), and Completely incorrect (1 point). A double-blinded design was employed to minimize bias. Statistical analyses, including t-tests, ANOVA, and correlation analyses, were used to compare scores, word counts, and response times.

Datasets

The study did not use pre-existing datasets for training LLMs. Instead, it created a custom evaluation dataset of 52 bone cancer-related questions based on the NCCN Bone Cancer Clinical Practice Guidelines (Version 2.2025). This dataset is specific to bone cancer and aims to assess LLM performance against expert-curated medical knowledge.

Key Insights and Ideas

  • High Overall Accuracy: All three LLMs demonstrated a remarkable ability to provide accurate information regarding bone cancer, with over 90% of their responses deemed correct (combining "Comprehensive" and "Correct but incomplete" categories). This suggests LLMs are becoming increasingly reliable for accessing foundational medical knowledge.
  • No Correlation Between Quality and Quantity/Speed: Surprisingly, the study found no significant correlation between the accuracy of an LLM's response, the number of words it used, or the time it took to generate the answer. This implies that longer or faster responses don't necessarily equate to better or worse accuracy.
  • Domain-Specific Nuances: While overall performance was strong, there were specific instances where LLMs showed variations. ChatGPT 5 and Grok 4 outperformed Deepseek V3.1 when answering questions specifically about Ewing sarcoma.
  • Potential for Patient Education: The paper highlights the potential of LLMs to enhance patient education by providing accessible, evidence-based information, which can improve treatment adherence and patient satisfaction.

Key Results

  • Accuracy: In the first round, the probabilities of providing correct responses were: Deepseek V3.1 (92.4%), ChatGPT 5 (93.3%), and Grok 4 (96.2%). In the second round, these figures slightly increased to 94.3%, 95.2%, and 97.1%, respectively.
  • Scores: The average total scores were very close: Deepseek V3.1 (3.75 ± 0.71), ChatGPT 5 (3.81 ± 0.6), and Grok 4 (3.87 ± 0.51), with no statistically significant differences overall. However, ChatGPT 5 scored significantly higher than Deepseek V3.1 for Ewing sarcoma questions.
  • Word Count: Deepseek V3.1 consistently produced the longest responses (546.56 ± 194.49 words), significantly more than ChatGPT 5 (367.02 ± 273.18) and Grok 4 (194.16 ± 197.07).
  • Response Time: ChatGPT 5 was the fastest (1.52 ± 0.52 seconds), while Grok 4 was the slowest (42.48 ± 26.89 seconds), with Deepseek V3.1 falling in between (11.83 ± 3.41 seconds).
  • Consistency: Scores remained consistent between the two response rounds for all LLMs. However, ChatGPT 5 showed a statistically significant difference in word count between rounds.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Runs and Language: The study used only two independent runs and a single language, which may not fully capture the stochastic variability and prompt sensitivity of LLMs. Future work should include multilingual, repeated tests.
  • General-Purpose LLMs: The study focused on general-purpose LLMs and did not include medically fine-tuned models, which might offer different performance characteristics.
  • Specific Discrepancies: Despite high overall accuracy, there were instances of incorrect answers, such as for Question 15 (adjuvant chemotherapy for Ewing sarcoma) and Question 21 (predilection site of periosteal osteosarcoma), where LLMs diverged from clinical guidelines.

Future research should focus on:

  • Developing open-access LLMs trained on real medical data to enhance reliability.
  • Exploring mechanisms to estimate and communicate uncertainty in LLM responses.
  • Establishing ethical frameworks and regulatory guidelines for LLM use in healthcare, particularly concerning responsibility for errors.

Summary updated

Abstract

When answering bone cancer-related questions, Deepseek V3.1, ChatGPT 5, and Grok 4 generally performed well. Specifically, when responding to questions about Ewing sarcoma, ChatGPT 5 and Grok 4 demonstrated higher accuracy than Deepseek V3.1. While each model has its own strengths and limitations, their collective potential to enhance medical knowledge and improve healthcare outcomes is undeniable.
BibTeX
@inproceedings{pan2025evaluation,
  author = {Qi Pan and Liping Huang and Ning Liu and Feng Lin and S. Ye},
  title = {{Evaluation of the accuracy of large language models in answering bone cancer-related questions}},
  booktitle = {Frontiers in Public Health},
  year = {2025},
  doi = {10.3389/fpubh.2025.1719047},
}
Figure