This study investigates how post-training techniques like reinforcement learning with verifiable rewards (RLVR) actually change the behavior of Vision-Language Models (VLMs) in the context of autonomous driving. By evaluating the Gemma model family on multi-view driving questions, the researchers demonstrate that while aggregate accuracy improves, these gains can mask dangerous regressions in safety-critical scenarios due to unweighted reward objectives.
TL;DR
The authors use Group Relative Policy Optimization (GRPO) to tune Gemma models for multi-view driving QA, finding that post-training gains primarily correct substantive answer choices rather than just fixing output formatting. However, they uncover a "safety asymmetry" where improvements in routine perception queries come at the cost of performance in rare, safety-critical hazard predictions.
What Problem Does This Paper Address?
When we "tune" a foundation model for a specific task like driving, we often see accuracy scores go up. But why are they going up? In structured tasks (like those requiring JSON output), a model might fail simply because it didn't follow the formatting instructions, even if it "knew" the right answer. Conversely, a model might get better at following instructions without actually improving its visual understanding of the road.
Furthermore, in autonomous driving, not all questions are created equal. Missing a "what color is that house?" query is trivial; missing a "is that pedestrian about to step into my path?" query is catastrophic. This paper investigates whether standard reinforcement learning objectives—which treat all correct answers as equal—create hidden risks by trading off safety for aggregate score.
Methods
The researchers focused on the Gemma-4 family of models, specifically a matched-harness comparison of a Gemma-4-12B base model against a version tuned using Group Relative Policy Optimization (GRPO).
Key methodological components included:
- Verifiable Rewards: Instead of using a subjective "reward model," they used rule-based verifiers to score outputs based on three criteria: Correctness (matching the ground truth), Format (strict JSON compliance), and Extractability (whether an answer could be found at all).
- QLoRA Tuning: The 12B model was tuned using QLoRA (rank 64) for 500 optimizer steps.
- Transition Analysis: They didn't just look at final scores; they tracked "wrong-to-correct" and "correct-to-wrong" transitions to see if the model was actually learning new information or just fixing its syntax.
Datasets
The study utilized WaymoQA, a dataset derived from the Waymo Open Dataset. It features:
- Multi-view observations: Synchronized inputs from eight cameras providing a panoramic view.
- Diverse Query Types: 5,519 test questions spanning perception, trajectory prediction, planning, and safety-critical interventions.
- Discrete Labels: Multiple-choice style answers that allow for objective, verifiable scoring.
Key Insights and Ideas
- Substance Over Syntax: A common critique of RLVR is that it just teaches models to "speak JSON." However, this study found that out of 215 improvements in the 12B model, 213 were cases where the base model already provided an extractable (but wrong) answer. This suggests the tuning actually shifted the model's decision-making policy, not just its formatting.
- The Safety Asymmetry: This is the paper's most critical finding. The model showed a +5.93 percentage point gain in "Normal Perception & Prediction" but suffered a -1.82 point regression in "Safety-critical Prediction." Because the reward function was unweighted, the optimizer found it easier to harvest gains from common routine scenes than to solve rare, difficult edge cases.
- Gradient Starvation: The authors observed "reward variance collapse." As the model mastered easy questions, the relative advantage between different sampled answers within a group disappeared, leading to zero gradient updates. This suggests that binary rewards are insufficient for mastering complex, long-tail distributions like driving.
- Thinking Isn't a Silver Bullet: In a separate test of a Gemma-31B model, allowing the model to "think" (generate internal reasoning tokens) before answering yielded no detected average improvement, suggesting that extra computation time doesn't automatically resolve spatial ambiguities in multi-camera setups.
Key Results
- Aggregate Gain: The Gemma-4-12B model's accuracy rose from 73.78% to 75.43% (+1.65 pp) after GRPO tuning.
- Zero-Shot Ladder: The study established a performance baseline across the Gemma family, with the 31B model reaching 77.21% accuracy.
- Category Divergence: As shown in the table below, gains were highly non-uniform. The model became much better at routine captioning and perception but worse at predicting hazards and following traffic signals.
Limitations and Future Work
- No SFT Control: The study lacks a direct comparison to Supervised Fine-Tuning (SFT), so it cannot claim GRPO is strictly superior to standard methods.
- Open-Loop Only: The evaluation is based on static questions (open-loop) and does not certify how these models would perform in a real, closed-loop driving environment where their actions affect the next state.
- Reward Weighting: The authors suggest that future work must move toward cost-sensitive weighting (e.g., penalizing safety errors more heavily than perception errors) to prevent the "safety-for-accuracy" trade-off observed here.
- Visual Grounding: The study notes that while answers improved, they cannot prove the model's visual understanding improved without further interventions like camera-ablation studies.
Summary updated