Abstract
Artificial intelligence (AI) has been deeply applied in the medical field and has shown broad application prospects. Pre-consultation system is an important supplement to the traditional face-to-face consultation. The combination of the AI and the pre-consultation system can help to raise the efficiency of the clinical work. However, it is still challenging for the AI to analyze and process the complicated electronic health record (EHR) data. Our pre-consultation system uses an automated natural language processing (NLP) system to communicate with the patients through the mobile terminals, applying the deep learning (DL) techniques to extract the symptomatic information, and finally outputs the structured electronic medical records. From November 2019 to May 2020, a total of 2,648 pediatric patients used our model to provide their medical history and get the primary diagnosis before visiting the physicians in the outpatient department of the Shanghai Children's Medical Center. Our task is to evaluate the ability of the AI and doctors to obtain the primary diagnosis and to analyze the effect of the consistency between the medical history described by our model and the physicians on the diagnostic performance. The results showed that if we do not consider whether the medical history recorded by the AI and doctors was consistent or not, our model performed worse compared to the physicians and had a lower average F1 score (0.825 vs. 0.912). However, when the chief complaint or the history of present illness described by the AI and doctors was consistent, our model had a higher average F1 score and was closer to the doctors. Finally, when the AI had the same diagnostic conditions with doctors, our model achieved a higher average F1 score (0.931) compared to the physicians (0.92). This study demonstrated that our model could obtain a more structured medical history and had a good diagnostic logic, which would help to improve the diagnostic accuracy of the outpatient doctors and reduce the misdiagnosis and missed diagnosis. But, our model still needs a good deal of training to obtain more accurate symptomatic information.
Introduction
Pediatric outpatient departments often face overwhelming patient volumes, leading to overworked doctors and dangerously short consultation windows. This environment increases the risk of misdiagnosis or missed symptoms. While AI has shown promise in medical imaging, applying it to the "front end" of a doctor's visit—the consultation—is difficult. The primary challenge is the "unstructured" nature of medical data; parents describe symptoms in plain, sometimes imprecise language. This paper evaluates whether an AI system can effectively bridge this gap by communicating with parents, structuring their input into Electronic Health Records (EHRs), and providing a preliminary diagnosis that rivals a human physician's accuracy.
Methods
The researchers developed a pre-consultation system that mimics the "reasoning-diagnosis" logic used by physicians. The workflow consists of an automated NLP interaction via a mobile interface, symptom extraction using deep learning to identify both positive and negative symptoms, and a diagnostic engine powered by the XGBoost algorithm trained on over 59,000 high-quality EHRs.
To understand why AI succeeds or fails, the researchers created a framework to evaluate the "consistency" of the medical history (Chief Complaint and History of Present Illness) between the AI's record and the doctor's record.
Datasets and Data Processing
The study utilized data from 2,648 pediatric patients at the Shanghai Children’s Medical Center (SCMC) collected between November 2019 and May 2020. After filtering for duplicates and incomplete records (e.g., patients visiting only for physicals), 1,573 pairs of medical records remained for analysis.
These records were subdivided into specific datasets to analyze the impact of data quality:
- Dataset 1: All 1,573 records.
- Dataset 2: Consistent Chief Complaint (n=935).
- Dataset 3: Consistent History of Present Illness (n=742).
- Dataset 4: Both consistent (n=536).
Key Results and Insights
The performance was measured using the F1 score (the harmonic mean of precision and recall). The study highlights that AI diagnostic failure is often a data acquisition problem, not a logic problem. When the AI was fed "noisy" or inconsistent history from parents, it performed poorly.
- General Performance (Dataset 1): Doctors outperformed the AI (0.912 vs. 0.825). This was attributed to parents providing inaccurate or "obscure" information to the AI.
- High-Consistency Performance (Dataset 4): When the medical history was consistent, the AI outperformed the doctors with an average F1 score of 0.931 vs. 0.920.
Superior Logic under Parity
The most striking insight is that when the AI and the doctor are working from the same set of facts (Dataset 4), the AI's diagnostic logic is actually more accurate and consistent than the human's. This suggests that the AI's "reasoning" is robust once the input data is standardized. In Dataset 4, the AI achieved particularly high scores in diagnosing Gastroenteritis (0.973), Mesenteric Lymphadenitis (0.976), and Urinary Tract Infections (0.962).
Discussion and Future Work
Unlike many basic symptom checkers, this system specifically extracts negative symptoms, which are crucial for differential diagnosis. The system successfully transformed unstructured parent descriptions into structured EHRs that doctors could directly cite, potentially saving significant time during visits.
However, the system's biggest weakness is its reliance on parents' ability to describe symptoms accurately. Future work will focus on adding visual aids to the mobile interface to help parents provide more accurate symptomatic information and training the model on "noisy" real-world data to improve its robustness.
Summary updated