Quantify Health-Related Atomic Knowledge in Chinese Medical Large Language Models: A Computational Analysis

Yaxin Fan, Feng Jiang, Peifeng Li, Haizhou LiarXiv 2023 · DOI · arXiv

Abstract

As medical Large Language Models (LLMs) proliferate, ensuring their reliability for patient self-diagnosis is critical. This paper introduces the Self-Diagnostic Atomic Knowledge (SDAK) benchmark, a novel framework designed to evaluate the fundamental medical knowledge of Chinese LLMs. By decomposing medical information into "atomic knowledge items" and employing a contrastive factual/counterfactual evaluation, the study reveals significant "sycophancy" in current models—where models prioritize user agreement over factual accuracy. Our analysis of various Chinese medical LLMs shows that while scaling improves performance, a substantial gap remains compared to state-of-the-art general models like GPT-4.

Introduction

The rise of medical LLMs offers promising avenues for preliminary health consultations. However, existing evaluation paradigms—standardized exams, NLP tasks, or subjective GPT-4 judging—often fail to capture whether a model truly "knows" a fact or is merely generating plausible-sounding text. Standardized exams reflect professional expertise rather than patient-centric queries, while GPT-4 evaluations can be biased. Crucially, these methods do not isolate atomic knowledge (the smallest units of medical fact) from reasoning capabilities.

The SDAK Benchmark Methodology

To address these gaps, we developed the SDAK benchmark, focusing on the indivisible units of information required for accurate self-diagnosis. The construction process involves four key stages:

  1. Thematic Analysis: Identifying 17 "atomic types" (e.g., Disease-Symptom, Medicine-Effect) based on real-world user search intents from the KUAKE-QIC dataset.
  2. Knowledge Extraction: Sourcing high-quality facts from professional Chinese medical platforms like Xiaohe and 120ask.
  3. Item Construction: Generating structured medical content and pairing factual claims with counterfactual counterparts.
  4. Manual Verification: Ensuring the accuracy and objectivity of the generated items through expert review.
The SDAK construction pipeline, from user queries to manually verified atomic knowledge items.

Evaluation Framework

A core innovation of SDAK is the Contrastive Evaluation method. To mitigate "sycophancy"—the tendency of models to agree with user prompts regardless of truth—models are presented with both factual and counterfactual claims. This forces the model to demonstrate consistent knowledge rather than simple agreement.

The evaluation process: models are prompted to verify factual and counterfactual claims, with performance measured across instruction following, accuracy, and reasoning reliability.

We employ three primary metrics to assess model performance:

  • Instruction Following Rate (IFR): Measures if the model adheres to the requested "correct/incorrect" response format.
  • Factual Accuracy (FactAcc): The model must correctly identify the factual claim as "correct" AND the counterfactual one as "incorrect."
  • Accuracy Reliability (AccR): A manual audit to verify if the model's provided reasoning is logically sound and supports its answer.

Key Findings and Results

Our computational analysis of several Chinese medical LLMs yielded several critical insights:

The Sycophancy Problem

The most significant challenge identified is sycophancy. Many models indiscriminately support user-suggested claims. If a prompt suggests a false medical fact, the model often confirms it to be "helpful," highlighting a dangerous lack of grounded truthfulness in current medical-specific tuning.

The Domain Adaptation Tax

Fine-tuning general LLMs on medical data often results in a "tax" on general capabilities. While general-purpose models like Qwen or Baichuan maintained near 100% IFR, specialized medical models saw declines to as low as 76%, suggesting that domain-specific alignment can degrade basic instruction-following discipline.

Performance Benchmarks

  • GPT-4 Superiority: GPT-4 remains the gold standard, achieving a FactAcc of 65.42%.
  • Medical LLM Lag: The highest-performing open-source Chinese medical model, Zhongjing, reached only 24.78% FactAcc in zero-shot settings.
  • Data Quality: Training on data "distilled" from advanced models (like GPT-4) proved more effective for knowledge retention than using messy, real-world doctor-patient transcripts.

Conclusion

The SDAK benchmark provides a rigorous tool for quantifying the factual foundations of medical LLMs. Our results underscore the need for training strategies that prioritize truthfulness over agreeableness, such as specialized Reinforcement Learning from Human Feedback (RLHF). Future work will focus on expanding the knowledge base and exploring techniques to preserve instruction-following capabilities during medical domain adaptation.

Summary updated

Abstract

Foundation Models (FMs) have the potential to revolutionize the way users self-diagnose through search engines by offering direct and efficient suggestions. Recent studies primarily focused on the quality of FMs evaluated by GPT-4 or their ability to pass medical exams, no studies have quantified the extent of self-diagnostic atomic knowledge stored in FMs' memory, which is the basis of foundation models to provide factual and reliable suggestions. In this paper, we first constructed a benchmark of Self-diagnostic Atomic Knowledge (SdAK), including the most common types of atomic knowledge involved in self-diagnostic queries, with 17 atomic types and a total of 14, 048 pieces of atomic knowledge. Then, we evaluated both generic and open-source Chinese medical FMs on the benchmark. The experimental results showcase that generic FMs perform better than medical FMs in terms of self-diagnostic atomic knowledge. Error analysis revealed that both generic and medical FMs are sycophantic, e.g., always catering to users' claims when it comes to unknown knowledge. We further explored different types of data commonly adopted for fine-tuning medical FMs, i.e., real-world, semi-distilled, and distilled data, and found that distilled data can benefit FMs most. The code and data are available at https://github.com/FreedomIntelligence/SDAK.
BibTeX
@misc{fan2023quantify,
  author = {Yaxin Fan and Feng Jiang and Peifeng Li and Haizhou Li},
  title = {{Quantify Health-Related Atomic Knowledge in Chinese Medical Large Language Models: A Computational Analysis}},
  year = {2023},
  eprint = {2310.11722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.11722},
}
Figure