Abstract
Interactive medical questioning is essential in real-world clinical consultations, where physicians must actively gather information from patients. While medical Large Language Models (LLMs) have shown impressive capabilities in static medical question answering, they predominantly operate under a reactive paradigm: generating answers directly without seeking additional information, which risks incorrect diagnoses in such interactive settings. To address this limitation, we propose ProMed, a reinforcement learning (RL) framework that transitions medical LLMs toward a proactive paradigm, equipping them with the ability to ask clinically valuable questions before decision-making. At the core of ProMed is the Shapley Information Gain (SIG) reward, which quantifies the clinical utility of each question by combining the amount of newly acquired information with its contextual importance, estimated via Shapley values. We integrate SIG into a two-stage training pipeline: (1) SIG-Guided Model Initialization uses Monte Carlo Tree Search (MCTS) to construct high-reward interaction trajectories to supervise the model, and (2) SIG-Augmented Policy Optimization, which integrates SIG and enhances RL with a novel SIG-guided Reward Distribution Mechanism that assigns higher rewards to informative questions for targeted optimization. Extensive experiments on two newly curated partial-information medical benchmarks demonstrate that ProMed significantly outperforms state-of-the-art methods by an average of 6.29% and delivers a 54.45% gain over the reactive paradigm, while also generalizing robustly to out-of-domain cases.
Introduction
Current medical Large Language Models (LLMs) typically follow a "reactive" paradigm, attempting to diagnose patients immediately upon receiving initial symptoms. However, clinical reality is interactive; initial patient descriptions are often vague or incomplete. A reactive approach in these scenarios frequently leads to misdiagnosis because the model lacks the critical information necessary to differentiate between similar conditions.
To bridge this gap, we introduce ProMed, a framework designed to train LLMs to be "proactive." ProMed enables models to recognize when information is insufficient and to actively seek missing clinical data through targeted questioning before committing to a diagnosis.
Methodology
The ProMed framework centers on a novel reward mechanism and a two-stage training pipeline.
Shapley Information Gain (SIG)
A fundamental challenge in proactive training is defining the value of a question. ProMed utilizes Shapley Information Gain (SIG), drawing from cooperative game theory. By treating medical facts as "players" contributing to a correct diagnosis, we calculate Shapley values to quantify the marginal contribution of each fact. This ensures that a question is rewarded not just for uncovering any information, but for uncovering clinically significant information that interacts synergistically with existing knowledge to improve diagnostic confidence.
Two-Stage Training Pipeline
- SIG-Guided Initialization: We employ Monte Carlo Tree Search (MCTS) to explore potential doctor-patient interaction trajectories. High-reward paths—those that maximize SIG and lead to correct diagnoses—are used for Supervised Fine-Tuning (SFT) to establish a strong proactive baseline.
- SIG-Augmented Policy Optimization: The model is further refined using a modified Group Relative Policy Optimization (GRPO) algorithm. We introduce a SIG-guided Reward Distribution Mechanism that assigns rewards to specific tokens within the dialogue. This fine-grained credit assignment allows the model to learn exactly which questions were most valuable, discouraging redundant or irrelevant inquiry.
Experimental Evaluation
We evaluated ProMed using two interactive benchmarks derived from MedQA (USMLE-based) and CMB (Chinese Medical Benchmark). Static questions were decomposed into "atomic facts" to facilitate a simulated doctor-patient dialogue environment.
Key Results
- Diagnostic Accuracy: ProMed achieved a 54.45% improvement over the reactive paradigm and outperformed state-of-the-art proactive methods by an average of 6.29%.
- Generalization: ProMed demonstrated robust cross-domain performance, maintaining its lead when trained on CMB and tested on MedQA, indicating the learning of a generalizable diagnostic strategy.
- Efficiency: Even at smaller scales (8B parameters), ProMed-optimized models outperformed larger specialized medical models like HuatuoGPT-o1 on interactive tasks.
Conclusion
ProMed successfully transitions medical LLMs from reactive guessers to proactive diagnostic agents. By leveraging Shapley values for information gain, the framework ensures that models prioritize clinically relevant data. Future work will focus on integrating multi-modal data (e.g., medical imaging) and extending these capabilities to long-term clinical reasoning.
Summary updated