SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation

Jinyu Li, Zhiming Zhang, Hao Fang, Hongyao Yu, Bin Chen, Albert Ma, Shu-Tao XiaarXiv 2026 · arXiv

TL;DR

SARGO is a post-training framework that mitigates visual hallucinations in Vision-Language Models (VLMs) by applying "token-level credit assignment." By using a model’s own self-guidance to distinguish between grounded facts and unsupported claims within the same sentence, it reduces errors without making the model overly cautious or uninformative.

What Problem Does This Paper Address?

Vision-Language Models (like Qwen or LLaVA) often suffer from "visual hallucinations"—claiming an object exists in an image when it clearly doesn't. Current mitigation strategies usually rely on Reinforcement Learning (RL), but they face a fundamental "blame assignment" problem.

Standard RL typically operates at the response level. If a model generates a long description that is 90% accurate but contains one hallucination, the entire response receives a negative reward. The model then learns to suppress everything in that response, including the correct information. Over time, this makes models "lazy" or overly brief; they learn that the safest way to avoid hallucinating is to say as little as possible, leading to poor object recall and reduced utility.

Methods

The authors introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), which moves from coarse response-level updates to surgical token-level updates.

  1. On-Policy Self-Guidance: During training, the model acts as its own teacher. It generates a response and then re-evaluates its own tokens while given "privileged context" (ground-truth object labels from the dataset). If a token’s probability increases significantly when the model "sees" the ground truth, that token is considered well-supported.
  2. Advantage Routing (The Agreement Gate): SARGO calculates two separate advantages: one for overall response quality ($A^{RL}$) and one specifically for hallucination reduction ($A^{hall}$). It only applies token-level guidance when these two signals agree. This prevents the model from accidentally reinforcing grounded tokens in a response that is otherwise poor (e.g., repetitive or nonsensical).
  3. Direct and Reciprocal Weighting: This is the core innovation.
  4. Positive Updates: It uses direct weighting to give more credit to well-supported tokens.
  5. Negative Updates: It uses reciprocal weighting to ensure the penalty falls primarily on the unsupported (hallucinated) tokens, while sparing the grounded ones from being suppressed.
  6. Reference-Calibrated Reward: To prevent the model from achieving a high score by simply omitting details, the reward function compares the new response to a baseline from a frozen version of the model. It specifically rewards "coverage expansion"—finding new, correct objects that the original model missed.

Datasets

The models (Qwen3-VL-4B/8B and Gemma3-4B) were trained on 10,000 images from COCO-Stuff, utilizing its rich grounding annotations (object labels and scene categories).

Evaluation was conducted across several specialized benchmarks:

  • Obj-Hal: Measures object-level hallucination and recall.
  • AMBER: Evaluates hallucination rate and information coverage.
  • THRONE: A probe-based benchmark for object presence.
  • SHR: A GPT-4-assisted metric for sentence-level hallucination ratios.
  • MME & MMStar: Used to ensure the model maintains general multimodal reasoning capabilities.

Key Insights and Ideas

  • The "Recall-Factuality" Trade-off is Not Inevitable: Most methods trade recall for factuality (saying less to be "safer"). SARGO shows that by protecting grounded tokens during negative updates, you can actually improve factuality while increasing the amount of useful information the model provides.
  • Reciprocal Weighting is Essential: Standard self-distillation methods often apply the same weights regardless of whether the update is positive or negative. SARGO’s insight is that the "teacher" should protect good tokens from being collateral damage when a bad response is penalized.
  • Hallucination-Aware Routing: By gating the guidance based on the agreement between the general reward and the hallucination reward, the authors ensure that the model doesn't get "confused" by responses that are factual but otherwise low-quality.

Key Results

SARGO consistently outperformed strong baselines (like OPA-DPO and SymMPO) across all tested backbones:

  • Hallucination Reduction: On Qwen3-VL-4B, SARGO reduced the object-level hallucination rate (CHAIRi) from 10.3% to 5.8%.
  • Information Retention: Unlike other methods that saw coverage drop, SARGO increased AMBER coverage from 74.2% (Vanilla) to 76.2%.
  • General Capabilities: Remarkably, SARGO improved general multimodal scores. On the MME benchmark, it achieved the highest scores for both Qwen3-VL scales, proving that fixing hallucinations doesn't have to come at the cost of general intelligence.

Limitations and Future Work

  • Annotation Dependency: The "teacher" phase requires high-quality grounding annotations (like COCO-Stuff). Scaling this to web-scale data where such annotations are missing remains a challenge.
  • Computational Cost: The framework requires a dual-pass (with and without privileged context) during the rollout phase of RL, which increases training time compared to standard PPO/GRPO.
  • Future Directions: The authors suggest exploring how this selective reinforcement could be applied to more complex multimodal reasoning tasks beyond simple object description.

Summary updated

Abstract

Vision-language models remain susceptible to visual hallucinations, often mixing grounded information with unsupported claims in the same response. Response-level reinforcement learning assigns a shared advantage to all tokens in a response, causing negative updates to suppress correct information alongside unsupported claims. To reduce hallucination while preserving useful visual information, we introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework for selective reinforcement and suppression through on-policy self-guidance. Direct and reciprocal weighting gives better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Agreement between overall and hallucination-specific advantages selects when to apply this guidance. At the response level, a reference-calibrated reward couples factuality with the retention and expansion of grounded information, discouraging lower hallucination scores obtained through omission. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate that SARGO outperforms the evaluated mitigation baselines across Obj-Hal, AMBER, THRONE, and SHR evaluations. SARGO achieves these gains while preserving useful visual information and general multimodal capabilities.
BibTeX
@misc{li2026sargo,
  author = {Jinyu Li and Zhiming Zhang and Hao Fang and Hongyao Yu and Bin Chen and Albert Ma and Shu-Tao Xia},
  title = {{SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation}},
  year = {2026},
  eprint = {2611.19388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19388},
}
Figure