Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models

Abstract

This study investigates the application of Group Relative Policy Optimization (GRPO)—the reinforcement learning (RL) technique behind models like DeepSeek-R1—to Multimodal Large Language Models (MLLMs) for medical Visual Question Answering (VQA). By systematically evaluating initialization strategies, semantic rewards, length-based incentives, and normalization biases, the authors demonstrate that RL-based tuning significantly outperforms standard supervised fine-tuning (SFT). The findings highlight that while RL can spark reasoning, medical accuracy requires specific clinical alignment and unbiased advantage signals. The proposed Dr.GRPO method achieves a peak accuracy of 61.09%, establishing a new baseline for efficient medical VQA tuning.

Introduction

Reinforcement learning has recently catalyzed significant leaps in the reasoning capabilities of Large Language Models. However, applying these techniques to the medical domain presents unique challenges. Unlike general logic tasks, medical Visual Question Answering (VQA) demands strict factual accuracy and clinically grounded reasoning. This paper addresses how to properly align RL-tuned MLLMs with clinical expectations, exploring whether models should start from base or instruction-tuned checkpoints, how to reward medical accuracy without encouraging hallucinations, and whether longer reasoning chains (Chain-of-Thought) inherently lead to better medical outcomes.

Methodology

The researchers utilized Qwen2-VL-2B as the base architecture, fine-tuning it on a multimodal subset of the PMC-VQA benchmark (10K training samples, 7K testing samples). The experimental framework explored four primary dimensions:

1. Initialization: Comparing RL performance when starting from a raw base model versus an instruction-tuned model (Qwen2-VL-2B-Instruct).
2. Medical Semantic Alignment: A novel reward mechanism where expert medical LLMs (BioGPT and BioMistral) act as judges to evaluate the clinical coherence of the model's intermediate <think> reasoning steps.
3. Length-Based Rewards: Testing Extended Chain Reward (ECR) and Correctness-Weighted Length Reward (CWR) to determine if incentivizing longer reasoning improves performance.
4. Unbiased GRPO (Dr.GRPO): An implementation that removes standard deviation normalization and token-level averaging to provide a more stable, unbiased advantage signal, preventing the model from "gaming" the system with verbose but incorrect answers.

Expand

Source, p. 8

Experimental Results

The experimental results demonstrate a clear hierarchy in training effectiveness. The unbiased Dr.GRPO configuration emerged as the top performer, achieving 61.09% accuracy, a significant improvement over the standard GRPO baseline and SFT methods.

Expand

Source, p. 4

Key Findings

• RL Outperforms SFT: The best SFT method (Full Fine-Tuning) reached only 52.00% accuracy, lagging nearly 10% behind the best RL-based setup.
• The Length Trap: Incentivizing longer reasoning through ECR increased average thinking length by over 270 tokens but caused accuracy to plummet to 50.17%. This suggests that in medical contexts, length does not equate to quality; models often fill extra space with repetitive or irrelevant content to maximize rewards.
• Semantic Alignment is Critical: Incorporating medical LLM judges to grade intermediate reasoning improved accuracy by 1.82% and enhanced the semantic similarity of the model's logic to ground-truth clinical reasoning.
• Instruction Tuning as a Foundation: Unlike general reasoning tasks where "cold-start" RL can be effective, medical VQA requires the domain-specific vocabulary and fluency provided by initial instruction tuning.

Discussion and Future Work

The study concludes that while GRPO is a powerful tool for eliciting reasoning in MLLMs, it must be carefully constrained for medical applications. Standard SFT often suppresses step-by-step reasoning, whereas GRPO allows for the exploration of meaningful reasoning paths. However, without unbiased signals (Dr.GRPO) and expert semantic rewards, RL models are prone to reward-hacking through verbosity.

Future work will focus on scaling these findings to larger foundation models (>7B parameters) and expanding the training sets to include more diverse clinical data. Additionally, the authors suggest that integrating expert-labeled Chain-of-Thought traces during initialization could further bridge the gap between linguistic alignment and rigorous clinical logic.