MedGR$^2$: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning

Weihai Zhi, Jiayan Guo, Shangyang Li

TL;DR

MedGR² breaks the chronic data scarcity bottleneck in medical AI by establishing a self-improving, closed-loop framework that co-develops a multimodal data generator and a dynamic reward model. By autonomously synthesizing and filtering high-quality medical reasoning data, this approach enables a compact 7B-parameter model to achieve state-of-the-art performance, outperforming foundation models ten times its size.

What Problem Does This Paper Address?

Deploying Vision-Language Models (VLMs) in clinical settings is severely limited by a fundamental bottleneck: the scarcity of high-quality, expert-annotated medical data. Acquiring this data is constrained by strict privacy regulations, immense annotation costs, and the rare nature of specialized medical expertise.

To circumvent this, researchers typically try two paths, both with major drawbacks:

  1. Supervised Fine-Tuning (SFT) on curated or LLM-cleaned biomedical text/image pairs. However, these models often memorize the training distribution and fail to generalize to unseen medical modalities or tasks.
  2. Reinforcement Learning (RL), which encourages robust reasoning strategies. Unfortunately, RL relies heavily on accurate reward signals, which traditionally require expensive human expert judgments or preference annotations to train.

This creates a paradox: RL is needed to improve generalization in data-scarce domains, but the supervision required to train the RL reward model is itself too costly to scale.

Methods

MedGR² reframes the problem from data curation to automated data generation and filtering via a three-stage, closed-loop ecosystem:

Overview of the MedGR² framework showing the three sequential stages: VQA Generation, Reward Model Alignment, and Two-Stage Policy Optimization.
  1. Stage 1: Multimodal Generator ($G_\theta$): Using a powerful backbone (Gemini-2.0-Flash), the generator synthesizes diverse Visual Question Answering (VQA) triplets (image, question, rationale-grounded answer). The authors utilize a meta-cognitive introspection prompt that forces the model to reflect on question types and expected structures, yielding highly diverse clinical outputs.
  2. Stage 2: Co-Evolving Reward Model ($R_\phi$): A lightweight VLM evaluates the generated triplets across four dimensions: factual accuracy, formatting accuracy, reasoning soundness, and instruction relevance. Instead of standard pairwise preferences, it uses multi-grade supervision trained on four distinct quality tiers (ranging from real radiologist reports to heavily corrupted/hallucinated text). Crucially, the reward model continually adapts and retrains on the evolving outputs of the reasoning policy.
  3. Stage 3: Two-Stage Policy Optimization ($\pi_\psi$):
  4. Warm Start: The target policy (Qwen2.5-VL-7B) undergoes SFT on the highest-scoring reward-filtered data to anchor it in a space of coherent reasoning.
  5. RL Refinement: The policy is optimized via Group Relative Policy Optimization (GRPO) using a hybrid dense-plus-sparse reward function that combines the reward model's semantic score with an exact entity-matching check.

Datasets

The framework was trained and evaluated primarily on:

  • OmniMedVQA: A massive, multi-specialty medical VQA benchmark containing over 300,000 triplets spanning eight distinct imaging modalities (CT, MRI, X-Ray, Ultrasound, Dermatology, Fundus Photography, OCT, and Microscopy).
  • MMMU: Used as an out-of-domain benchmark to test the model's broader generalization capabilities.

Key Insights and Ideas

  • Proactive Generation Beats Passive Curation: Rather than trying to clean noisy, static web corpora (like PubMed), proactively generating data from scratch under the strict guidance of a multi-grade reward model yields superior training fidelity.
  • Extreme Data Efficiency: SFT using only 2,000 high-reward, filtered samples (TopK) outperformed a baseline SFT trained on 11,000 raw, unfiltered samples.
Figure 2: The synergy of reward-filtered data and Reinforce- ment Learning across varying data scales. We compare four training strategies: ‘RandK‘ (SFT on K randomly sampled generated data); ‘TopK‘ (SFT on the top K samples se- lected by our reward model; ‘Rand11000‘ (a baseline SFT on 11,000 unfiltered samples); and ‘TopK+Rand‘ (MedGR2 with GRPO applied on the ‘TopK‘ data).
  • The RL Sweet Spot: The authors discovered that RL (GRPO) provides the highest marginal utility when the volume of available high-quality data is small to moderate (1k–4k samples). At larger scales (8k+), SFT-only models catch up, suggesting a saturation point for imitation learning.
  • Systematic Error Correction: Head-to-head transition analysis proved that MedGR² doesn't just memorize frequencies; it actively corrects baseline errors across both high-resource (X-Ray) and low-resource (Dermatology) modalities.
Head-to-Head Error Transition Analysis showing how MedGR² systematically corrects baseline errors across multiple modalities.

Key Results

  • State-of-the-Art Performance: The full MedGR² (SFT+GRPO) model achieved an overall accuracy of 87.45% on OmniMedVQA.
  • Defeating Giants: The 7B-parameter MedGR² model achieved a +19.74% absolute gain over the massive, 72B-parameter Qwen2.5-VL-72B foundation model.
  • Modality Dominance: The framework achieved exceptional top-tier accuracy in highly heterogeneous domains, including 92.65% in X-Ray, 95.55% in Fundus Photography, and 94.42% in OCT.
  • Robust Generalization: On the out-of-domain MMMU benchmark, the full pipeline achieved 78.66% accuracy, outperforming the raw SFT baseline by 5.48%.

Limitations and Future Work

While MedGR² demonstrates remarkable parameter and data efficiency, the paper notes or implies a few limitations:

  • Seed Dependency: The framework still requires a small initial seed dataset (e.g., 2k preference pairs distilled from GPT-4o) to bootstrap the reward model.
  • API Reliance: The data generation engine relies on commercial multimodal APIs (like Gemini), which may introduce upstream biases or availability constraints.
  • RL Saturation: Because the benefits of RL diminish as the volume of high-quality synthetic data scales up to 8k+ samples, future work is needed to optimize RL exploration strategies for massive synthetic datasets.

Summary updated

Abstract

The application of Vision-Language Models (VLMs) in medicine is critically hampered by the scarcity of high-quality, expert-annotated data. Supervised Fine-Tuning (SFT) on existing datasets often leads to poor generalization on unseen modalities and tasks, while Reinforcement Learning (RL), a promising alternative, is stymied by the lack of reliable reward signals in this data-scarce domain. To break this impasse, we introduce Generative Reward Learning for Medical Reasoning (MedGR$^2$), a novel framework that creates a self-improving virtuous cycle. MedGR$^2$ co-develops a data generator and a reward model, enabling the automated, continuous creation of high-quality, multi-modal medical data that serves as both a superior training source for SFT and RL. Our experiments demonstrate that SFT with MedGR$^2$-produced data already surpasses baselines trained on large-scale, human-curated datasets. Crucially, when leveraging this data for RL via Group Relative Policy Optimization (GRPO), our model achieves state-of-the-art cross-modality and cross-task generalization, significantly outperforming specialized RL-based methods. Furthermore, our compact model, empowered by MedGR$^2$, achieves performance competitive with foundation models possessing over 10 times more parameters. MedGR$^2$ presents a new paradigm for data-efficient learning in high-stakes domains, transforming the problem from data scarcity to data generation and unlocking the full potential of RL for building truly generalizable medical AI.
BibTeX
@inproceedings{zhi2026medgr,
  author = {Weihai Zhi and Jiayan Guo and Shangyang Li},
  title = {{MedGR$^2$: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=QScSLAP0Ou},
}
Figure