Large Language Models (LLMs) routinely achieve passing scores on standardized medical licensing examinations. However, when deployed at the bedside, these same models frequently struggle. This discrepancy highlights a fundamental gap between static knowledge retrieval and the dynamic, interactive reality of clinical practice.
To bridge this gap, a major research direction has emerged that reframes medical AI evaluation and architecture. Instead of treating diagnosis as a one-shot classification task where all information is known upfront, this paradigm models clinical reasoning as a sequential decision-making process driven by interactive inquiry.
The Evaluation Illusion of Static Benchmarks
Traditional medical AI benchmarks rely on static clinical vignettes—paragraphs of text that cleanly summarize a patient's history, physical exam, and lab results. The model's job is simply to output the correct diagnostic label.
Freya Gulamali characterizes this phenomenon as an "evaluation illusion" in [The evaluation illusion of large language models in medicine]. Static multiple-choice questions or single-turn prompts pre-package all necessary context, bypassing the most challenging phase of real-world medicine: information elicitation. In a live clinic, patients present with vague, unorganized, or ambiguous symptoms. The clinician must actively interview the patient, clarify contradictions, and decide which questions to ask or tests to order.
When models are forced to transition from passive summaries to active information gathering, their performance often degrades. As shown by Friederike Jungmann et al. in [Evaluation and mitigation of the limitations of large language models in clinical decision-making], models evaluated on raw patient cases struggle to autonomously gather information or systematically adhere to diagnostic guidelines. Furthermore, Nikita Mehandru et al. note in [Evaluating large language models as agents in the clinic] that static tests fail to account for how errors can cascade across a sequence of clinical actions, meaning high exam scores do not guarantee real-world safety.
Diagnosis as a Sequential Decision Process
To mimic human clinical intuition, automated systems must treat diagnosis not as a destination, but as a continuous trajectory of meaning-making. This concept aligns with the perspective of Kevin W. Eva et al. in [What’s in a Label? Is Diagnosis the Start or the End of Clinical Reasoning?], who argue that overemphasizing static diagnostic labels encourages premature cognitive closure.
Mathematically and algorithmically, this is achieved by framing clinical inquiry as a sequential decision process. Early work by Yuan Ling et al. in [Learning to Diagnose: Assimilating Clinical Narratives using Deep Reinforcement Learning] demonstrated that modeling clinical narratives incrementally allows systems to better handle noise and weigh evidence step-by-step.
[Initial Vague Symptom]
│
▼
┌───────────────────────────┐
│ Formulate Hypotheses │◀─┐
└───────────────────────────┘ │
│ │ Iterative
▼ │ Feedback
┌───────────────────────────┐ │ Loop
│ Select Next Inquiry/Test │ │
└───────────────────────────┘ │
│ │
▼ │
┌───────────────────────────┐ │
│ Observe New Information │──┘
└───────────────────────────┘
│
▼
[Differential Diagnosis]
By utilizing Reinforcement Learning (RL), an agent can be trained to balance the value of new information against the "cost" (such as time, money, or patient discomfort) of obtaining it. For example, Victor Ardulov et al. applied Q-learning in [Robust diagnostic classification via Q-learning] to build a diagnostic system that actively requests the most informative symptoms while penalizing excessive data collection, making the system highly robust to missing data.
In modern LLM architectures, this sequential logic is enforced through multi-agent frameworks or specialized prompting strategies:
- Hypothesis-Driven Testing: In [Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learning], David Bani-Harouni et al. introduced an agent framework that sequentially forms hypotheses and selects cost-effective medical tests while explicitly verbalizing its internal uncertainty.
- Incremental Verification: To prevent models from relying on eliminative shortcuts, Saeel Sandeep Nachane et al. developed CLINICR in [Few shot chain-of-thought driven reasoning to prompt LLMs for open-ended medical question answering]. This prompting strategy forces the model to follow a forward-backward verification path, updating its differential diagnosis step-by-step as new findings emerge.
- Modular History-Taking: The authors of [MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis] split the diagnostic task into distinct modules—such as automated history-taking and external knowledge retrieval—turning a one-shot prediction into an explainable, multi-turn dialogue.
Simulating the Bedside: Interactive Benchmarks
If clinical AI must act as an autonomous agent, evaluation frameworks must transition from static datasets to interactive simulations. Researchers are increasingly using task-oriented dialogue systems to act as "Standardized Patients" that reveal symptoms only when prompted correctly.
Ruoyu Liu et al. introduced MMD-Eval in [Interactive Evaluation for Medical LLMs via Task-oriented Dialogue System], providing a hallucination-free environment to measure an LLM's proactivity in gathering information during multi-turn consultations. Similarly, [Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models] by Christopher Chiu et al. leverages the structure of oral medical exams (viva voce) to force models to navigate diagnostic trajectories from ambiguous starting points, exposing human-like cognitive flaws such as anchoring bias.
This shift toward interactive evaluation also allows researchers to measure how well models handle real-world clinical messiness, such as demographic biases and systemic ambiguity. Max Lamparth et al. developed MENTAT [Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare], a psychiatric dataset designed to test language models within multi-turn interactions where patient data evolves dynamically.
By evaluating the entire interactive trajectory rather than just the final answer, these frameworks ensure that medical AI systems are safe, thorough, and capable of managing the open-ended uncertainties of actual clinical practice.
Key Foundational Work
Dynamic Network Updating Techniques for Diagnostic Reasoning (1991) — contributed 1 concept elements to this direction.
What’s in a Label? Is Diagnosis the Start or the End of Clinical Reasoning? (2016) — contributed 1 concept elements to this direction.
Implementing Troubleshooting with Batch Repair (2016) — contributed 1 concept elements to this direction.