Abstract
Background: The quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain. Methods: Anonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text. Results: SMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, P<0.001; 92.00 vs 85.80%, P<0.001; 85.60 vs 45.60%, P<0.001; 89.60 vs 45.40%, P<0.001). Physicians’ rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, P<0.001; 2.53 vs 2.33, P<0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity, readability, and logicality. AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, P = 0.008) than human-generated EMRs. Conclusions: The quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation.
Introduction
Outpatient consultations are high-pressure environments where physicians often face fatigue and high patient volumes. These conditions frequently lead to inconsistent or incomplete Electronic Medical Records (EMRs). Such deficiencies are more than just administrative issues; they compromise patient safety, increase the risk of misdiagnosis, and elevate medical malpractice concerns. While general Large Language Models (LLMs) like GPT-4 show promise, they often struggle with "hallucinations" and lack the domain-specific precision required for nuanced clinical details, such as the specific onset characteristics of abdominal pain. This study introduces SMART-ASSISTANT, a specialized AI system designed to provide real-time assistance in history taking and EMR composition.
Methods
The SMART-ASSISTANT architecture comprises four integrated pillars:
- Audio Transcription: Utilizing FunASR with custom entity correction to accurately capture medical terminology from consultation dialogues.
- Structured EMR Generation: Summarizing free-text dialogue into standardized medical sections like Chief Complaint and History of Present Illness.
- Quality Control (QC): Identifying "omission points" based on standardized templates to ensure critical clinical questions are addressed.
- Assisted Diagnosis: Providing a list of the most likely diagnoses based on extracted data.
The system was developed using LoRA (Low-Rank Adaptation) to fine-tune open-source LLMs (HuatuoGPT-II, Llama-3, and ChatGLM-3) on specialized medical datasets, including 1,249 retrospective EMR cases and 119 simulated free-text/structured pairs.
Results
The fine-tuned models demonstrated significant advantages over general-purpose LLMs. Specifically, HuatuoGPT-II-FT achieved 100% accuracy in symptom identification, vastly outperforming GPT-4's 19.20%. In retrospective tests, the fine-tuned model reached an overall quality control point extraction accuracy of 91.53%, compared to GPT-4's 71.61%.
In real-world prospective trials, the system maintained high fidelity, achieving a mean Semantic Textual Similarity (STS) of 0.9253. Clinical validation through a multi-reader multi-case study showed that physicians rated AI-assisted records as significantly more complete (4.27 vs 3.92) and diagnostically correlated than human-generated records. Furthermore, the introduction of a "Theoretical Completion Rate" (TCR) suggested that human-AI collaboration could improve record completeness from roughly 73% to nearly 98% by addressing real-time flags for missing information.
Discussion and Conclusion
SMART-ASSISTANT successfully demonstrates that domain-specific fine-tuning can overcome the limitations of general LLMs in clinical settings. By acting as a real-time "safety net," the system ensures that critical diagnostic questions are not overlooked during busy outpatient sessions.
However, challenges remain, including acoustic interference in noisy clinics and the current focus on a single symptom (abdominal pain). Future work will aim to expand the system's scope to a broader range of outpatient complaints and validate its performance in multi-center, international settings. Despite these hurdles, the study confirms that SMART-ASSISTANT significantly improves EMR quality, offering substantial benefits for both patient care and healthcare organizational efficiency.
Summary updated