Master clinical medical knowledge at certificated-doctor-level with deep learning model

Ji Wu, Xien Liu, Xiao Zhang, Zhiyang He, Ping LvNature Communications 2018 · DOI

Abstract

Mastering of medical knowledge to human is a lengthy process that typically involves several years of school study and residency training. Recently, deep learning algorithms have shown potential in solving medical problems. Here we demonstrate mastering clinical medical knowledge at certificated-doctor-level via a deep learning framework Med3R, which utilizes a human-like learning and reasoning process. Med3R becomes the first AI system that has successfully passed the written test of National Medical Licensing Examination in China 2017 with 456 scores, surpassing 96.3% human examinees. Med3R is further applied for providing aided clinical diagnosis service based on real electronic medical records. Compared to human experts and competitive baselines, our system can provide more accurate and consistent clinical diagnosis results. Med3R provides a potential possibility to alleviate the severe shortage of qualified doctors in countries and small cities of China by providing computer-aided medical care and health services for patients.

Introduction

While artificial intelligence has achieved significant success in narrow medical tasks, such as specialized image recognition, developing a general-purpose clinical AI remains a significant challenge. Human physicians synthesize vast amounts of unstructured text, clinical experience, and logical reasoning to diagnose diverse conditions. This paper introduces Med3R, a deep learning framework designed to bridge the gap between pattern matching and clinical reasoning, aiming to address the shortage of qualified medical professionals in underserved regions.

The Med3R Framework

Med3R employs a three-stage learning architecture that mimics the human medical education path:

  1. Free Reading (Unsupervised Learning): The model processes a massive corpus of medical textbooks to create embeddings that capture seven specific medical relationships, such as the association between symptoms and diseases.
  2. Guided Reading (Supervised Learning): Using a large-scale medical Q&A dataset (MedQA), the model refines its knowledge into "delta-embeddings." This allows the system to distinguish between subtle clinical variations, such as the nuances between different types of nephritis.
  3. Multi-layer Reasoning: The model utilizes three distinct reasoning layers—Keypoint Reasoning (KR), Context Reasoning (CR), and Global Reasoning (GR)—to fuse specific symptoms with broader clinical contexts.

Experimental Results

National Medical Licensing Examination (NMLEC)

Med3R was evaluated using the 2017 Chinese National Medical Licensing Examination. The system achieved a score of 456, significantly exceeding the passing threshold of 360. This performance placed the model in the top 3.7% of all human examinees for that year.

Score distribution of the 2017 NMLEC, showing Med3R surpassing 96.3% of human examinees with a score of 456.

Benchmarking and Clinical Performance

In comparative testing, Med3R consistently outperformed existing state-of-the-art systems, including WatsonQA-like architectures. Across seven different practice tests, Med3R maintained a mean accuracy of approximately 78%, compared to the ~68% achieved by baseline models.

Accuracy comparison between Med3R and WatsonQA across seven test sets and the mean performance.

Furthermore, when applied to 15,000 real-world Electronic Medical Records (EMR) covering 50 diseases, Med3R achieved a diagnostic accuracy of 92.04%. The system demonstrated higher consistency than human experts, who may be subject to fatigue or subjective bias.

Discussion and Conclusion

Med3R represents a milestone in medical AI, proving that deep learning can master clinical knowledge at a level comparable to certified doctors. While the model currently focuses on written data and does not yet incorporate physical examinations or visual patient cues, its success in reasoning through complex clinical scenarios is unprecedented. The researchers envision Med3R as a computer-aided diagnostic tool to support general practitioners in rural areas, improving healthcare accessibility across China.

Summary updated

Abstract

Mastering of medical knowledge to human is a lengthy process that typically involves several years of school study and residency training. Recently, deep learning algorithms have shown potential in solving medical problems. Here we demonstrate mastering clinical medical knowledge at certificated-doctor-level via a deep learning framework Med3R, which utilizes a human-like learning and reasoning process. Med3R becomes the first AI system that has successfully passed the written test of National Medical Licensing Examination in China 2017 with 456 scores, surpassing 96.3% human examinees. Med3R is further applied for providing aided clinical diagnosis service based on real electronic medical records. Compared to human experts and competitive baselines, our system can provide more accurate and consistent clinical diagnosis results. Med3R provides a potential possibility to alleviate the severe shortage of qualified doctors in countries and small cities of China by providing computer-aided medical care and health services for patients.
BibTeX
@inproceedings{wu2018master,
  author = {Ji Wu and Xien Liu and Xiao Zhang and Zhiyang He and Ping Lv},
  title = {{Master clinical medical knowledge at certificated-doctor-level with deep learning model}},
  booktitle = {Nature Communications},
  year = {2018},
  doi = {10.1038/s41467-018-06799-6},
}
Figure