GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

Wanqi Dong, Wanyue Dong, Jingze Ge, Mehul MotaniarXiv 2026 · arXiv

TL;DR

GRASP is a novel post-training recipe for medical Visual Question Answering (VQA) that simultaneously combines Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) in every update. By supervising the gold answer using the model's own generated reasoning as context—and adaptively weighting this supervision based on group accuracy—GRASP allows small models (8B parameters) trained on modest datasets (12.9K samples) to outperform massive medical VLMs on out-of-domain benchmarks.

What Problem Does This Paper Address?

Adapting general-purpose vision-language models (VLMs) to the medical domain is notoriously difficult due to two main constraints:

  1. Scarce and "Thin" Data: Medical datasets are small due to privacy and expert costs. Furthermore, they typically only contain short "gold" answers (e.g., "Pneumonia") without the reasoning traces (Chain-of-Thought) that modern models need to excel.
  2. The Generalization Trade-off: Standard Supervised Fine-Tuning (SFT) is fast but prone to overfitting and "eroding" the base model's general knowledge, leading to poor performance on out-of-domain (OOD) images from different hospitals or devices. Reinforcement Learning (RL) generalizes better but struggles when the model can't find the right answer on its own; if every sampled response in a group is wrong, algorithms like GRPO receive no signal to move toward the correct answer.

Methods

The authors propose GRASP (Group-adaptive Reinforcement with Answer Supervision on self-generated reasoning Paths). Unlike previous hybrids that stage SFT and RL or switch between them, GRASP applies both in a single update step.

Figure 1: GRASP: simultaneous RL and SFT training on the same rollouts. For each question with only a short gold answer, the policy samples 𝐺=8 reason-then-answer rollouts whose group accuracy is 𝑝. Top (RL): group-relative advantages plus an adaptive extreme-group correction 𝑐(𝑝) for all-wrong and all-correct groups drive the GSPO-EGC loss. Bottom (SFT): each rollout’s own reasoning is kept as a loss-free prefix, followed by the teacher-forced gold answer, weighted by 𝜆(𝑝). Policy: Qwen3-VL-8B with frozen base weights; both losses update only the LoRA adapters in the ViT and the LLM, and no reasoning annotations are used.

Key components include:

  • Simultaneous Hybrid Objective: For every question, the model generates $G=8$ "reason-then-answer" rollouts. The RL branch (GSPO-EGC) learns from the relative quality of these rollouts, while the SFT branch provides direct supervision toward the gold answer.
  • Supervision on Own Reasoning: Instead of needing expert reasoning traces, GRASP uses the model's own generated reasoning as the context for SFT. The model is trained to predict the gold answer after its own reasoning path, which prevents it from learning to skip reasoning entirely.
  • Group-Adaptive Weighting ($\lambda(p)$): The strength of the SFT signal is inversely proportional to the group's accuracy. If the model already gets most rollouts right, SFT fades away, letting RL refine the reasoning. If the model is totally lost (all-wrong group), SFT provides the necessary "north star."
  • Extreme-Group Correction (EGC): Standard group-relative RL fails when all samples are equally right or wrong. EGC adaptively shifts the advantage based on the model's running accuracy, ensuring a training signal even in these "extreme" cases.

Datasets

The authors introduce MedTerse, a curated dataset of 12.9K short-answer questions derived from PathVQA, SLAKE, VQA-RAD, and GMAI-MMBench.

For evaluation, they use:

  • In-domain: Test splits from the source datasets.
  • Out-of-domain (OOD): A newly converted version of OmniMedVQA featuring 11.6K questions across eight imaging modalities (CT, MRI, X-ray, etc.).
  • External sets: Kvasir-VQA (endoscopy) and VQA-Med 2020 (diagnoses).
Figure 2: Data. (a) MedTerse: 12,893 short-answer training questions from four public sources. (b) 11,575 OOD test questions over eight OmniMedVQA modali- ties. Inner rings: share of yes/no questions.

Key Insights and Ideas

  • Reasoning as Context, Not Target: A major "aha" moment is that you don't need to supervise the reasoning itself to improve performance. Using the model's own (potentially flawed) reasoning as a prefix for the gold answer provides enough context to help the model map visual features to the correct medical term.
  • The "Gold Injection" Pitfall: The authors found that simply adding the gold answer to the RL rollout group (as in some prior work) causes the model to "cheat" by learning to skip reasoning to maximize reward. GRASP avoids this by keeping the SFT and RL branches distinct but concurrent.
  • Efficiency is Possible: By tuning only ~1% of parameters (via LoRA), GRASP-trained models beat medical VLMs that were trained on up to 2,600 times more data. This suggests that the quality and method of post-training matter more than raw scale in specialized domains.

Key Results

GRASP consistently outperformed all baselines across three different backbones (Qwen3-VL-8B, InternVL3-8B, and Gemma 3-4B).

  • OOD Superiority: On the OmniMedVQA benchmark, GRASP-trained InternVL3-8B achieved 27.3% Exact Match (EM), significantly beating the best existing medical VLM (Lingshu-7B at 18.5%) and general-purpose models.
  • In-Domain Performance: GRASP achieved 78.2% EM on the SLAKE dataset, outperforming pure SFT (71.3%) and standard RL (59.8%).
  • Generalization: As shown in Figure 3, the simultaneous hybrid approach provides a massive boost over zero-shot performance, particularly on the smaller Gemma 3-4B backbone (+31.7 points in-domain).
Figure 3: Performance gains of GRASP over zero-shot and other objectives across different backbones.

Limitations and Future Work

  • Reasoning-Answer Consistency: While GRASP improves accuracy, the model occasionally generates reasoning that contradicts its final answer. The authors suggest "gating" supervision based on consistency as a future direction.
  • Short-Answer Constraint: The current work is limited to answers of 1–3 words. Extending this to long-form clinical reports remains a challenge.
  • Evaluation Metrics: While the authors used an LLM judge and human physician audits to supplement Exact Match, medical VQA still lacks a perfect automated metric that captures clinical nuance.

Summary updated

Abstract

Adapting general-purpose vision-language models (VLMs) to medical visual question answering (VQA) is valuable for helping clinicians read images and make diagnoses, but annotated data are scarce and provide only short answers, without reasoning annotations. Supervised fine-tuning (SFT) fits these labels but overfits to them and amplifies their biases, eroding out-of-domain (OOD) generalization. Reinforcement learning (RL) better preserves generalization but never uses the gold answer as a training target: GRPO-style algorithms learn from the contrast between correct and incorrect answers within a group of sampled responses, so all-correct or all-wrong groups give no correctness signal. Existing SFT-RL hybrids either apply SFT and RL in separate stages or switch between them per question, or need reasoning traces from experts or stronger models, which medical VQA data lack. We propose GRASP, which combines RL and SFT in every update using only short answers: RL lets the model explore its own reasoning, SFT teaches the gold answer after that reasoning to supply the direct supervision RL lacks, and their balance adapts to how well the model answers each question. Tuning only about 1% of the parameters with LoRA, GRASP outperforms all compared training objectives both in domain and on OOD open-ended questions across Qwen3-VL-8B, InternVL3-8B and Gemma 3-4B. On an OOD benchmark spanning eight imaging modalities, GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared general and medical VLMs in open-ended accuracy, including models trained on up to 2,600 times more data or with over 250 times more trainable parameters. We release MedTerse, a dataset of 12.9K short-answer questions, and our code at https://github.com/sea-comet/GRASP-MedVQA.
BibTeX
@misc{dong2026grasp,
  author = {Wanqi Dong and Wanyue Dong and Jingze Ge and Mehul Motani},
  title = {{GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations}},
  year = {2026},
  eprint = {2610.34561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34561},
}
Figure