Abstract: Vladimir Shaposhnikov, Alexandr Nesterov, Ilia Kopanichuk, Ivan Bakulin, Zhelvakov Egor, Ruslan Abramov, Tsapieva Ekaterina Olegovna, Iaroslav Radionovich Bespalov, Dmitry V. Dylov, Ivan Oseledets. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track.
2025.
Introduction
The primary bottleneck in modern healthcare often begins at the point of entry. Patients typically must consult a General Practitioner (GP) as an intermediary before being referred to a specialist, leading to unnecessary delays, increased costs, and administrative burdens. While Large Language Models (LLMs) offer a potential solution for automating these initial consultations, they face significant hurdles in clinical settings: they are prone to hallucinations, struggle to maintain a structured diagnostic flow, and often fail to reliably identify life-threatening emergencies. This paper introduces CLARITY to bridge the gap between the flexibility of conversational AI and the rigid safety requirements of medical practice.
System Architecture
CLARITY is built on a hybrid architecture that avoids the "black box" unpredictability of a pure LLM approach. The system utilizes a modular microservices framework governed by a deterministic Finite State Machine (FSM).
Key components include:
- Chat Manager (The Skeleton): An FSM that governs the dialogue flow across six contexts (e.g., Initialization, Info Collection, Emergency). This ensures the conversation follows a logical, safe, and predictable path.
- Decision-Making Services: Includes an Emergency Detector (using HGB and LLM embeddings), a Readiness Estimator to determine when to provide a referral, and a Moderator to filter unsafe topics.
- Text Generation Services: Powered by GigaChat, these modules generate context-aware questions and diagnostic hypotheses.
- Information Collector: Uses a vector database to search for similar historical cases to reuse high-quality questions, reducing latency.
Evaluation and Results
The authors evaluated CLARITY using a mix of internal training sets and massive real-world deployment data, including over 55,000 content-rich user dialogues from a national interhospital platform. To validate routing accuracy, 2,500 dialogues were manually annotated by licensed doctors.
CLARITY achieved 77% precision for the first-attempt specialist recommendation (Precision@1) and 96% recall for the top three recommendations (R@3). Operationally, the average consultation took just 2 minutes and 13 seconds, making it three times faster than traditional human-led sessions. Furthermore, 32.4% of users who completed a dialogue converted to an actual medical appointment, indicating high user trust.
Key Insights
- FSM as a Safety Rail: LLMs in healthcare should not "drive" the conversation; they should "power" steps defined by a deterministic FSM. This prevents the AI from skipping critical safety checks like emergency screening.
- The Efficiency-Satisfaction Paradox: While CLARITY was significantly faster than humans, 31% of users felt the bot asked "too few" questions, suggesting that the perception of thoroughness is as vital as diagnostic accuracy in medicine.
- Scalability via Hierarchy: A multi-level FSM design allows the system to expand to new medical domains linearly rather than exponentially, avoiding architectural brittleness.
Limitations and Future Work
While the FSM provides structure, the internal reasoning of the LLM remains difficult to inspect. Future iterations plan to include Chain-of-Thought (CoT) tracing for better clinician transparency. Additionally, the system is currently validated primarily on common clinical cases; expanding the pipeline to include synthetic data for rare diseases and complex multi-morbidity cases remains a priority.
Summary updated