Answer Correction with Verifiable Rewards: A Gemma-Family Study of Multi-View Driving QA

T N Rickesh, Pranav Vinodh, Sowmya Kamath S

This study investigates how post-training techniques like reinforcement learning with verifiable rewards (RLVR) actually change the behavior of Vision-Language Models (VLMs) in the context of autonomous driving. By evaluating the Gemma model family on multi-view driving questions, the researchers demonstrate that while aggregate accuracy improves, these gains can mask dangerous regressions in safety-critical scenarios due to unweighted reward objectives.

TL;DR

The authors use Group Relative Policy Optimization (GRPO) to tune Gemma models for multi-view driving QA, finding that post-training gains primarily correct substantive answer choices rather than just fixing output formatting. However, they uncover a "safety asymmetry" where improvements in routine perception queries come at the cost of performance in rare, safety-critical hazard predictions.

What Problem Does This Paper Address?

When we "tune" a foundation model for a specific task like driving, we often see accuracy scores go up. But why are they going up? In structured tasks (like those requiring JSON output), a model might fail simply because it didn't follow the formatting instructions, even if it "knew" the right answer. Conversely, a model might get better at following instructions without actually improving its visual understanding of the road.

Furthermore, in autonomous driving, not all questions are created equal. Missing a "what color is that house?" query is trivial; missing a "is that pedestrian about to step into my path?" query is catastrophic. This paper investigates whether standard reinforcement learning objectives—which treat all correct answers as equal—create hidden risks by trading off safety for aggregate score.

Methods

The researchers focused on the Gemma-4 family of models, specifically a matched-harness comparison of a Gemma-4-12B base model against a version tuned using Group Relative Policy Optimization (GRPO).

Key methodological components included:

  • Verifiable Rewards: Instead of using a subjective "reward model," they used rule-based verifiers to score outputs based on three criteria: Correctness (matching the ground truth), Format (strict JSON compliance), and Extractability (whether an answer could be found at all).
  • QLoRA Tuning: The 12B model was tuned using QLoRA (rank 64) for 500 optimizer steps.
  • Transition Analysis: They didn't just look at final scores; they tracked "wrong-to-correct" and "correct-to-wrong" transitions to see if the model was actually learning new information or just fixing its syntax.

Datasets

The study utilized WaymoQA, a dataset derived from the Waymo Open Dataset. It features:

  • Multi-view observations: Synchronized inputs from eight cameras providing a panoramic view.
  • Diverse Query Types: 5,519 test questions spanning perception, trajectory prediction, planning, and safety-critical interventions.
  • Discrete Labels: Multiple-choice style answers that allow for objective, verifiable scoring.

Key Insights and Ideas

  • Substance Over Syntax: A common critique of RLVR is that it just teaches models to "speak JSON." However, this study found that out of 215 improvements in the 12B model, 213 were cases where the base model already provided an extractable (but wrong) answer. This suggests the tuning actually shifted the model's decision-making policy, not just its formatting.
  • The Safety Asymmetry: This is the paper's most critical finding. The model showed a +5.93 percentage point gain in "Normal Perception & Prediction" but suffered a -1.82 point regression in "Safety-critical Prediction." Because the reward function was unweighted, the optimizer found it easier to harvest gains from common routine scenes than to solve rare, difficult edge cases.
  • Gradient Starvation: The authors observed "reward variance collapse." As the model mastered easy questions, the relative advantage between different sampled answers within a group disappeared, leading to zero gradient updates. This suggests that binary rewards are insufficient for mastering complex, long-tail distributions like driving.
  • Thinking Isn't a Silver Bullet: In a separate test of a Gemma-31B model, allowing the model to "think" (generate internal reasoning tokens) before answering yielded no detected average improvement, suggesting that extra computation time doesn't automatically resolve spatial ambiguities in multi-camera setups.
This figure shows that while strict compliance improved, the vast majority of test items were already extractable in the base model, meaning the gains were largely due to better answer selection.

Key Results

  • Aggregate Gain: The Gemma-4-12B model's accuracy rose from 73.78% to 75.43% (+1.65 pp) after GRPO tuning.
  • Zero-Shot Ladder: The study established a performance baseline across the Gemma family, with the 31B model reaching 77.21% accuracy.
  • Category Divergence: As shown in the table below, gains were highly non-uniform. The model became much better at routine captioning and perception but worse at predicting hazards and following traffic signals.
Table 3 highlights the divergence between routine gains and safety-critical regressions.

Limitations and Future Work

  • No SFT Control: The study lacks a direct comparison to Supervised Fine-Tuning (SFT), so it cannot claim GRPO is strictly superior to standard methods.
  • Open-Loop Only: The evaluation is based on static questions (open-loop) and does not certify how these models would perform in a real, closed-loop driving environment where their actions affect the next state.
  • Reward Weighting: The authors suggest that future work must move toward cost-sensitive weighting (e.g., penalizing safety errors more heavily than perception errors) to prevent the "safety-for-accuracy" trade-off observed here.
  • Visual Grounding: The study notes that while answers improved, they cannot prove the model's visual understanding improved without further interventions like camera-ablation studies.

Summary updated

Abstract

Multi-camera driving question answering provides a discrete test of vision-language models, but post-training gains can reflect changed selected answers or improved answer extraction. We investigate this distinction within the Gemma model family using archived, task-linked WaymoQA evaluations. A zero-shot diagnostic on 5,519 test questions spans effective-small, mixture-of-experts, unified, and dense checkpoints, with serving differences disclosed rather than interpreted as architecture effects. The strongest controlled evidence is a matched-harness Gemma-4-12B base/GRPO pair: accuracy rises from 73.78% to 75.43%, a gain of 1.65 percentage points with a scene-cluster bootstrap interval of [+0.84, +2.41]. There are 215 improvements and 124 regressions; 213 improvements start from an extractable but incorrect base answer, and only two recover previously unextractable responses. Crossing correctness with extraction state shows why correct noncompliant outputs cannot be counted as prediction errors. Category counts reveal a critical safety asymmetry: routine perception gains (+5.93 pp) mask a regression in safety-critical hazard prediction (-1.82 pp), demonstrating that unweighted outcome rewards can trade off rare edge cases for aggregate accuracy. Reward saturation describes sampled training dynamics rather than an architectural ceiling. A separate archived Gemma-31B thinking comparison yields no detected average improvement, with uncertainty that does not establish equivalence. These results support a modest answer-selection gain for the recorded tuning configuration and a reproducible diagnostic accounting, without attributing corrected labels to specific grounding mechanisms or claiming superiority over matched supervised tuning.
BibTeX
@misc{rickesh2026answer,
  author = {T N Rickesh and Pranav Vinodh and Sowmya Kamath S},
  title = {{Answer Correction with Verifiable Rewards: A Gemma-Family Study of Multi-View Driving QA}},
  year = {2026},
  eprint = {2611.00828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00828},
}
Figure