Evaluating the Challenges of LLMs in Real-world Medical Follow-up: A Comparative Study and An Optimized Framework

Jinyan Liu, Zikang Chen, Qinchuan Wang, Tan Xie, Heming Zheng, Xudong LvarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) show promise in medical dialogue, but "end-to-end" applications in structured medical follow-ups often result in logic errors, repetitive questioning, and excessive token consumption. We present a modular framework that decomposes the follow-up process into structured stages, including semantic clustering and rule-driven flow control. Our evaluation demonstrates that this approach reduces dialogue turns by approximately 47% and slashes token consumption by over 80% while maintaining high accuracy across complex clinical forms.

Introduction

Medical follow-up is a high-stakes task requiring adherence to complex forms with nested logic and conditional skip patterns. While LLMs excel at natural conversation, they struggle to autonomously manage these rigid structures. In standard end-to-end configurations, models frequently suffer from information loss, uncontrolled dialogue flow (e.g., infinite loops), and significant operational inefficiency. This study bridges the gap between LLM flexibility and clinical reliability by introducing a scaffolded, modular architecture.

Proposed Modular Framework

The proposed framework moves the "logic" of the follow-up out of the LLM's prompt and into a structured program. The system is divided into five key components:

  1. Form Structure Construction: Paper-based clinical forms are converted into structured JSON with explicit logic.
  2. Question Preprocessing: An LLM semantically clusters similar questions to streamline the interview.
  3. Question Generation: Empathetic, human-like queries are generated based on these clusters.
  4. RAG-based Intent Extraction: Retrieval-Augmented Generation (RAG) maps patient responses back to structured data using few-shot examples.
  5. Dialogue Flow Control: A rule-driven module manages transitions, ensuring no questions are missed or repeated.
The modular framework architecture, illustrating the transition from basic information to virtual patient interaction (a), the follow-up chatbot components (b), and the detailed dialogue management flow (c).

Experimental Setup

The framework was tested against an end-to-end LLM control group using 32 real-world medical follow-up forms covering common cancers. Three benchmark forms of varying complexity were used:

  • Form-1 (Simple): 10 single-choice questions.
  • Form-2 (Medium): 45 single-choice questions.
  • Form-3 (High Complexity): 53 questions with complex skip logic and multiple-choice/fill-in-the-blank types.

Evaluations were conducted using Qwen-plus, DeepSeek-chat, and GPT-4, interacting with "Virtual Patients" designed with diverse conversational traits.

Results and Discussion

The modular approach significantly outperformed the end-to-end baseline in efficiency, cost, and stability.

Efficiency and Cost Reduction

The modular framework reduced average dialogue turns by 46.73%. More strikingly, token consumption was 5 to 8 times lower than the control group. In complex scenarios, end-to-end models often spiraled into redundant loops, whereas the modular system maintained a tight, purposeful flow.

Comparison of token consumption between the control group (green) and the experimental modular group (orange) across different form complexities.

Accuracy and Stability

While all models performed well on simple forms, the accuracy of end-to-end models plummeted on complex forms (Form-3) due to logic failures. The modular system maintained stable accuracy by offloading bookkeeping to the rule-based controller. It also eliminated common failure modes like premature termination and infinite loops.

Conclusion

LLMs are not "plug-and-play" for tasks requiring strict logical adherence. By using LLMs for language processing and rule-based systems for flow control, we can create robust, efficient clinical tools. Future work will focus on reducing rule dependency through better generalization and incorporating multimodal inputs like voice and images to improve patient accessibility.

Summary updated

BibTeX
@misc{liu2025evaluating,
  author = {Jinyan Liu and Zikang Chen and Qinchuan Wang and Tan Xie and Heming Zheng and Xudong Lv},
  title = {{Evaluating the Challenges of LLMs in Real-world Medical Follow-up: A Comparative Study and An Optimized Framework}},
  year = {2025},
  eprint = {2512.18999},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18999},
}
Figure