TL;DR
Reinforcement Learning (RL) from automated "world feedback" is a promising way to train clinical agents, but it fails when benchmarks allow "silent-finish" shortcuts or require agents to guess obscure medical codes. This paper introduces MedAgentBench-v3 (MAB-v3) to fix these structural flaws and demonstrates that while RL is great for learning clinical logic, it requires Supervised Fine-Tuning (SFT) to handle the "format-knowledge" of complex medical identifiers.
What Problem Does This Paper Address?
The goal is to build AI agents that can execute clinical protocols—for example, checking a patient's potassium level and placing a replacement order if it's too low. Ideally, we should train these agents using RL from "world feedback": a rule-based verifier (the "world") grades the agent's actions, allowing for unlimited, automated training without expensive human labeling.
However, the authors found that applying RL to existing clinical benchmarks (MedAgentBench v1/v2) failed spectacularly. They identified two main culprits:
- The "Silent-Finish" Shortcut: In many tasks, doing nothing (a "silent finish") resulted in a passing grade because the patient didn't actually need an intervention. RL models quickly learned that "inaction" was the dominant strategy to maximize rewards, leading to 0% success on tasks that actually required action.
- The Format-Knowledge Barrier: Clinical tasks often require specific codes (like SNOMED or NDC). You cannot "explore" your way to a 15-digit ID number; if the agent doesn't already know the code, the reward landscape is flat, providing no gradient for the model to learn.
Methods
The authors performed a rigorous audit and "diagnosis" of RL in clinical environments:
- Benchmark Correction: They built MAB-v3, which fixes branch imbalances (ensuring most tasks require action) and documents previously "hidden" formatting requirements.
- Training Framework: They trained a Qwen3-8B model using GRPO (Group Relative Policy Optimization) for RL and compared it against SFT (Supervised Fine-Tuning) using programmatically generated "teacher" demonstrations.
- Reward Shaping: They implemented a reward function that provides partial credit for correct resource types but penalizes "spurious" actions to force the model to learn conditional logic.
- Taxonomy of Learnability: They categorized tasks into Decision (logic-based), Lookup (retrieval-based), and Format-knowledge (code-based) to predict which tasks RL can actually solve.
Datasets
- MedAgentBench-v3 (MAB-v3): A newly constructed benchmark of 508 tasks across 20 task types, involving ~100 real anonymized patients.
- Environment: A self-contained world feedback loop using an offline HAPI FHIR server with over 700,000 data elements.
- Comparison: The authors also evaluated several "frontier" models (GPT-4o, Gemini 1.5 Pro, etc.) on the new benchmark to establish a baseline.
Key Insights and Ideas
- RL is a "Reward Hacker": If a benchmark has a high "silent-finish ceiling" (where doing nothing earns points), RL will find that path and stay there. MAB-v3 lowered this ceiling from 41.7% to 8.9%, forcing the model to actually engage with the tools.
- The "Flat Landscape" Problem: RL is excellent at learning when to act (the decision logic), but it is "structurally insufficient" for learning what code to use (format knowledge). If a model doesn't know the SNOMED code for Naloxone from its pre-training, no amount of pure RL will help it "discover" it.
- SFT + RL is the Prescription: The authors argue that SFT and RL are complementary. SFT should be used to "inject" the necessary codes and formats, while RL should be used to refine the conditional reasoning and clinical logic.
Key Results
- Benchmark Integrity: MAB-v3 successfully closed the "inaction" shortcut, allowing RL to actually attempt clinical actions.
- The Performance Gap: Pure RL reached only 18.2% success, while rule-based SFT reached 34.1%.
- Diagnosis of the Gap: The 15.9 percentage point gap between SFT and RL was entirely attributable to "format-knowledge" and "capability-ceiling" failures.
- Frontier Models: Even top-tier models like GPT-5.5 and Gemini 3.1 Pro struggled with specific format-knowledge tasks (scoring 0% on some), despite high overall performance (~78%).
Limitations and Future Work
- Scope: The study focuses on "protocol execution" (administrative workflows) rather than open-ended clinical reasoning or novel medical judgments.
- Sample Size: Some specific task types in the 1:1 balanced split have very few instances, making per-type estimates somewhat noisy.
- Next Steps: The authors identify an SFT warmstart followed by RL as the "direct next experiment" to combine the strengths of both approaches. They also suggest extending this framework to broader, more complex clinical protocols.
Summary updated