Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study

Hongliu Du, Jialing Li, Bing Xiao, Xueying Wang, Wenxin Xue, Mei Deng, Sean Tan, Jiamin Wang, Chaijie Luo, Boru Chen, Ting Yang, Zhan Chen, Qinxuan Zu, Joseph J.�y. Sung, Honggang YuResearch Square 2025 · DOI

Abstract

Background: The quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain. Methods: Anonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text. Results: SMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, P<0.001; 92.00 vs 85.80%, P<0.001; 85.60 vs 45.60%, P<0.001; 89.60 vs 45.40%, P<0.001). Physicians’ rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, P<0.001; 2.53 vs 2.33, P<0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity, readability, and logicality. AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, P = 0.008) than human-generated EMRs. Conclusions: The quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation.

Introduction

Outpatient consultations are high-pressure environments where physicians often face fatigue and high patient volumes. These conditions frequently lead to inconsistent or incomplete Electronic Medical Records (EMRs). Such deficiencies are more than just administrative issues; they compromise patient safety, increase the risk of misdiagnosis, and elevate medical malpractice concerns. While general Large Language Models (LLMs) like GPT-4 show promise, they often struggle with "hallucinations" and lack the domain-specific precision required for nuanced clinical details, such as the specific onset characteristics of abdominal pain. This study introduces SMART-ASSISTANT, a specialized AI system designed to provide real-time assistance in history taking and EMR composition.

Methods

The SMART-ASSISTANT architecture comprises four integrated pillars:

  1. Audio Transcription: Utilizing FunASR with custom entity correction to accurately capture medical terminology from consultation dialogues.
  2. Structured EMR Generation: Summarizing free-text dialogue into standardized medical sections like Chief Complaint and History of Present Illness.
  3. Quality Control (QC): Identifying "omission points" based on standardized templates to ensure critical clinical questions are addressed.
  4. Assisted Diagnosis: Providing a list of the most likely diagnoses based on extracted data.

The system was developed using LoRA (Low-Rank Adaptation) to fine-tune open-source LLMs (HuatuoGPT-II, Llama-3, and ChatGLM-3) on specialized medical datasets, including 1,249 retrospective EMR cases and 119 simulated free-text/structured pairs.

Performance of audio and structured EMR transcription modules across different datasets and subjects.

Results

The fine-tuned models demonstrated significant advantages over general-purpose LLMs. Specifically, HuatuoGPT-II-FT achieved 100% accuracy in symptom identification, vastly outperforming GPT-4's 19.20%. In retrospective tests, the fine-tuned model reached an overall quality control point extraction accuracy of 91.53%, compared to GPT-4's 71.61%.

Comparison of model performance in retrospective EMR quality control point extraction across various clinical key points.

In real-world prospective trials, the system maintained high fidelity, achieving a mean Semantic Textual Similarity (STS) of 0.9253. Clinical validation through a multi-reader multi-case study showed that physicians rated AI-assisted records as significantly more complete (4.27 vs 3.92) and diagnostically correlated than human-generated records. Furthermore, the introduction of a "Theoretical Completion Rate" (TCR) suggested that human-AI collaboration could improve record completeness from roughly 73% to nearly 98% by addressing real-time flags for missing information.

Discussion and Conclusion

SMART-ASSISTANT successfully demonstrates that domain-specific fine-tuning can overcome the limitations of general LLMs in clinical settings. By acting as a real-time "safety net," the system ensures that critical diagnostic questions are not overlooked during busy outpatient sessions.

However, challenges remain, including acoustic interference in noisy clinics and the current focus on a single symptom (abdominal pain). Future work will aim to expand the system's scope to a broader range of outpatient complaints and validate its performance in multi-center, international settings. Despite these hurdles, the study confirms that SMART-ASSISTANT significantly improves EMR quality, offering substantial benefits for both patient care and healthcare organizational efficiency.

Summary updated

Abstract

<title>Abstract</title> Background The quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain. Methods Anonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text. Results SMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, <italic>P</italic>&lt;0.001; 92.00 vs 85.80%, <italic>P</italic>&lt;0.001; 85.60 vs 45.60%, <italic>P</italic>&lt;0.001; 89.60 vs 45.40%, <italic>P</italic>&lt;0.001). Physicians’ rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, <italic>P</italic>&lt;0.001; 2.53 vs 2.33, <italic>P</italic>&lt;0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity (3.23 vs 3.25, <italic>P</italic> = 0.806), readability (3.22 vs 3.38, <italic>P</italic> = 0.068), and logicality (3.19 vs 3.32, <italic>P</italic> = 0.122). AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, <italic>P</italic> = 0.008) than human-generated EMRs. Conclusions The quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation.
BibTeX
@inproceedings{du2025constructing,
  author = {Hongliu Du and Jialing Li and Bing Xiao and Xueying Wang and Wenxin Xue and Mei Deng and Sean Tan and Jiamin Wang and Chaijie Luo and Boru Chen and Ting Yang and Zhan Chen and Qinxuan Zu and Joseph J.�Y. Sung and Honggang Yu},
  title = {{Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7066667/v1},
}
Figure