Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning

Chi Liu, Derek Li, Yan Shu, Mengzhuo Chen, Derek Duan, Teng Fang, Bryan DaiSubmission to ICLR 2026 · Withdrawn

TL;DR

Fleming-R1 introduces a verifiable medical reasoning framework that pairs knowledge-graph-guided synthetic data with a two-stage reinforcement learning regime (RLVR). By training models to think iteratively rather than just predict answers, the 7B variant outperforms traditional 72B baselines, while the 32B model reaches near-parity with GPT-4o on the challenging MedXpertQA benchmark.

What Problem Does This Paper Address?

Large language models (LLMs) often achieve strong scores on multiple-choice medical benchmarks, but their clinical utility is severely limited by "answer without justification." In clinical decision-making, an unfaithful or opaque chain of thought is dangerous—a correct diagnostic guess supported by flawed pathophysiological reasoning can lead to fatal treatment decisions.

Existing models struggle with clinical reasoning due to three main issues:

  1. Static, shallow training data: Most medical datasets feature standard questions with sparse rationale supervision, largely ignoring rare diseases, niche medications, and multi-step trade-offs.
  2. Outcome-only optimization: Standard reinforcement learning optimizes for the correct final answer, failing to penalize intermediate diagnostic leaps or dosage errors.
  3. Fragile initialization: Without structured guidance at cold start, reasoning schemas easily collapse on out-of-distribution or compositionally complex cases.

Methods

Fleming-R1 addresses these challenges through a three-stage training pipeline:

  1. Reasoning-Oriented Data Strategy (RODS):
  2. Combines curated public medical question-answering datasets with synthetic data generated from an autonomous Wikipedia-derived medical knowledge graph (>100,000 entities).
  3. Samples multi-hop paths (e.g., symptom $\rightarrow$ pathophysiology $\rightarrow$ diagnostic test $\rightarrow$ treatment) focusing deliberately on underrepresented conditions. Plausible-but-wrong distractors are procedurally synthesized via relation-preserving perturbations.
  4. Chain-of-Thought (CoT) Cold Start:
  5. Rather than simple supervised fine-tuning, the authors distill reasoning trajectories from a high-capacity teacher model (GPT-OSS-120B).
  6. Uses an iterative refinement loop featuring backtracking, path exploration, and self-correction to build reasoning priors that check assumptions and correct intermediate errors.
  7. Two-Stage RLVR with GRPO:
  8. Employs Group Relative Policy Optimization (GRPO) using verifiable rewards strictly based on final answer accuracy and output formatting (avoiding length bias).
  9. Stage 1: Consolidates core reasoning on a balanced mix of easy and moderate cases.
  10. Stage 2: Employs adaptive hard-sample mining to dynamically target persistent model failure modes on difficult questions requiring multi-step inference.

Datasets

  • Training: Public medical QA corpora (MedQA, MedMCQA, CMExam, PubMedQA) combined with topologically sampled subgraphs from a curated Wikipedia knowledge graph.
  • Evaluation: Evaluated across nine medical benchmarks:
  • MedXpertQA (Text): Complex clinical scenarios spanning 17 specialties and 11 body systems.
  • Standard Medical Exams & QA: MedQA (USMLE), MedMCQA, Medbullets, PubMedQA, and CareQA (Spanish healthcare exam questions translated/rephrased).
  • Clinical Consultation & Domain Understanding: JMED (21-option doctor-patient hospital dialogue benchmark) and MMLU-Pro subsets (Biology and Health).

Key Insights and Ideas

  • Reasoning priors matter more than brute-force scaling: The 7B model initialized with self-correcting CoT and reinforced with RLVR outperforms several general-purpose and medical 32B/72B baselines.
  • Curriculum-driven hard negative mining prevents reward sparsity: Transitioning from easy/moderate problems to an adaptive curriculum focused on repeated failures prevents reinforcement learning from plateauing when dealing with difficult multi-hop diagnostic cases.
  • Knowledge graph masking creates rigorous clinical reasoning: By extracting subgraphs and masking key intermediate nodes (like lab results or treatments), models are forced to articulate disambiguating clinical evidence rather than rely on shallow surface cues.

Key Results

  • Fleming-R1-7B:
  • Achieved a 63.37% benchmark average, outperforming HuatuoGPT-O1-7B (56.12%) and Qwen2.5-7B (55.43%).
  • Surpassed the larger Qwen2.5-32B (63.40% vs. 63.37% essentially tied overall, but beating it on MedQA: 75.10% vs. 71.56%, and Medbullets: 57.05% vs. 48.99%).
  • Scored 19.14% on MedXpertQA, substantially outperforming Qwen2.5-7B-Instruct (11.96%) and beating the 72B Qwen2.5-72B (18.90%).
  • Fleming-R1-32B:
  • Achieved a 75.42% benchmark average, leading all tested open-source models in the 10B–100B tier (ahead of Qwen3-32B at 72.89% and HuatuoGPT-O1-72B at 72.45%).
  • Reached 30.33% on MedXpertQA, achieving near parity with closed-source GPT-4o (30.37%).

Limitations and Future Work

  • Evaluation is primarily multiple-choice: While benchmarks like MedXpertQA and JMED (21 options) stress reasoning, evaluation is constrained to structured MCQs rather than open-ended, multi-turn doctor-patient dialogues.
  • Slight regressions on dialogue tasks: On the 21-option JMED benchmark, the 32B model exhibited a small performance dip (-0.6 pp) during Stage 2 RL, indicating that hard-sample mining on exam-style questions may occasionally cause minor domain fluctuations.
  • Future Directions: Deploying verifiable reasoning into interactive clinical decision support systems and supporting real-time compliance auditing.

Summary updated

Abstract

While large language models show promise in medical applications, achieving expert-level clinical reasoning remains challenging due to the need for both accurate answers and transparent reasoning processes. To address this challenge, we introduce Fleming-R1, a model designed for verifiable medical reasoning through three complementary innovations. First, our Reasoning-Oriented Data Strategy (RODS) combines curated medical QA datasets with knowledge-graph-guided synthesis to improve coverage of underrepresented diseases, drugs, and multi-hop reasoning chains. Second, we employ Chain-of-Thought (CoT) cold start to distill high-quality reasoning trajectories from teacher models, establishing robust inference priors. Third, we implement a two-stage Reinforcement Learning from Verifiable Rewards (RLVR) framework using Group Relative Policy Optimization, which consolidates core reasoning skills while targeting persistent failure modes through adaptive hard-sample mining. Across diverse medical benchmarks, Fleming-R1 delivers substantial parameter-efficient improvements: the 7B variant surpasses much larger baselines, while the 32B model achieves near-parity with GPT-4o and consistently outperforms strong open-source alternatives. These results demonstrate that structured data design, reasoning-oriented initialization, and verifiable reinforcement learning can advance clinical reasoning beyond simple accuracy optimization. We release Fleming-R1 publicly to promote transparent, reproducible, and auditable progress in medical AI, enabling safer deployment in high-stakes clinical environments.
BibTeX
@misc{liu2025fleming,
  author = {Chi Liu and Derek Li and Yan Shu and Mengzhuo Chen and Derek Duan and Teng Fang and Bryan Dai},
  title = {{Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning}},
  year = {2025},
  eprint = {2509.15279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.15279},
}
Figure