Evolution of the “Internet Plus Health Care” Mode Enabled by Artificial Intelligence: Development and Application of an Outpatient Triage System

Lingrui Yang, Jiali Pang, Shu-Qing Zuo, Jian Xu, Jin Wei, Zuo Feng, Kui Xue, Zhongzhou Xiao, Xinwei Peng, Jie Xu, Xiaofan Zhang, Ruiyao Chen, Shuqing Luo, Shaoting Zhang, Xin SunJournal of Medical Internet Research 2024 · DOI

TL;DR

This paper introduces an AI-powered outpatient triage system designed for Chinese tertiary hospitals, addressing the challenge of patients struggling to navigate complex departmental structures. The system leverages BERT for accurate subspecialty department recommendations, significantly reducing patient registration cancellations and consultation times compared to traditional methods.

What Problem Does This Paper Address?

In China's large tertiary hospitals, patients often face difficulties identifying the correct specialized outpatient department due to a lack of medical knowledge and the sheer volume of departments and subspecialties. Existing online triage systems are often manual, prone to errors, and struggle with the nuances of Chinese patient descriptions. This leads to wasted time, resources, and a suboptimal patient experience, including misdiagnosis and treatment delays. The paper aims to solve this by developing a precise, AI-driven system that can recommend appropriate departments based on patient chief complaints, even for highly specialized areas.

Methods

The core of the system is built upon the BERT (Bidirectional Encoder Representations from Transformers) framework, adapted for Chinese language understanding. The development involved three key modules:

  1. Patient Description Extraction: This module identifies key medical events like symptoms and anatomical locations from patient descriptions.
  2. Patient Description Normalization: This crucial step converts non-standard patient descriptions into standardized medical terminologies, using the International Classification of Diseases, Injuries and Causes of Death (ICD-10) to reduce ambiguity.
  3. Department Recommendation: This module suggests appropriate departments based on the normalized medical concepts, along with patient age and gender.

To improve the model's adaptability to Chinese oral expressions and the complexity of chief complaints, the researchers employed a downsampling strategy. This involved training the model on EMR chief complaints but randomly selecting symptoms to simulate the less detailed nature of patient-generated descriptions.

Datasets

The study utilized two primary data sources:

  • Electronic Medical Records (EMRs): A large dataset of 395,790 EMRs from Xinhua Hospital was used for training, testing, and validation. These EMRs contained chief complaints, present medical histories, and past medical histories.
  • Medical Dialogue Groups: 500 medical dialogue groups were collected to help the model understand and normalize patient language.

The EMR data was split into:

  • Training data: 98% (n=387,876)
  • Testing data: 1% (n=3957)
  • Validation data: 1% (n=3957)

Key Insights and Ideas

  • BERT Adaptation for Chinese: Modifying the BERT framework was essential to handle the specific linguistic nuances and variations in Chinese patient descriptions, a significant hurdle in previous AI healthcare applications in China.
  • Downsampling EMRs for Realism: Training the model with downsampled EMR chief complaints, mimicking the brevity and potential vagueness of patient-generated text, proved more effective than using the full, detailed EMR data for certain performance metrics. This highlights the importance of simulating real-world input conditions.
  • Subspecialty Focus is Key: The system's ability to recommend specific subspecialty departments, rather than just broad categories, is a major advantage over human triage nurses or general physicians, leading to more precise referrals.
  • Bridging the "Internet Plus Health Care" Gap: The system directly addresses the "three long, one short" problem (long waiting times, long registration lines, long queues, short consultation times) by enabling more efficient and accurate prehospital triage, a critical step in advancing China's "Internet Plus Health Care" initiative.

Key Results

  • Accuracy: The triage system achieved an accuracy of 89.45% on the downsampled testing data set and 89.41% on the downsampled validation data set. In a prospective study, the system achieved 91.53% accuracy.
  • Cancellation Rate Reduction: Implementation in Xinhua Hospital led to a reduction in patient registration cancellations from 3.83% (2021) to 3.53% (2022).
  • Time Efficiency: The system significantly outperformed triage nurses in time consumption, taking an average of 10.11 seconds compared to 14.33 seconds for nurses.
  • Recall@3: The system demonstrated a higher recall@3 rate (62.30%) compared to triage nurses (52.66%), indicating it recommends more correct departments within the top three suggestions.
  • Comparison to Triage Nurses: While triage nurses showed higher precision (92.13% vs. 90.49%) and accuracy (98.03% vs. 91.53%), the AI system excelled in recall@3 and time efficiency, particularly in recommending subspecialty departments.

Limitations and Future Work

  • Limited Training Data: EMRs were collected from a single center (Xinhua Hospital), which might have a more specialized department structure than other hospitals. This could limit the generalizability of the model.
  • BERT vs. GPT-4: The study used BERT, which is effective for classification and extraction. However, the authors suggest that newer models like GPT-4, with their generative capabilities, could further enhance patient interaction and automatic medical record filling.
  • Early Promotion Phase: The system is still in its early stages, and its long-term impact on actual convenient medical service needs further investigation as user adoption grows.
  • Comparison Bias: The comparison with triage nurses had inherent biases, as nurses communicated directly, while the system relied on rephrased chief complaints. Future work should involve more precise labeling for fairer comparisons.

The authors envision the system being integrated into internet hospitals to provide convenient medical services and further evolve the "Internet Plus Health Care" model. They also suggest collaborative efforts for a more comprehensive and universal triage system.

Summary updated

Abstract

The triage system demonstrates high accuracy in outpatient triage of all departments and excels in subspecialty department recommendations, which could decrease the cancellation rate and time consumption. It also improves the efficiency and convenience of clinical care to fulfill better the usage of medical resources, expand hospital effectiveness, and improve patient satisfaction in Chinese tertiary hospitals.
BibTeX
@article{yang2024evolution,
  author = {Lingrui Yang and Jiali Pang and Shu-Qing Zuo and Jian Xu and Jin Wei and Zuo Feng and Kui Xue and Zhongzhou Xiao and Xinwei Peng and Jie Xu and Xiaofan Zhang and Ruiyao Chen and Shuqing Luo and Shaoting Zhang and Xin Sun},
  title = {{Evolution of the “Internet Plus Health Care” Mode Enabled by Artificial Intelligence: Development and Application of an Outpatient Triage System}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/51711},
}
Figure