TL;DR
GEM-MM introduces an entropy-guided approach to aligning Vision-Language Models (VLMs), moving away from "entropy-blind" methods like standard DPO that treat every token as equally important. By rewarding uncertainty during reasoning ("fork tokens") but penalizing it in final answers, GEM-MM significantly reduces visual hallucinations on a tight data budget.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) are notorious for "hallucinating"—confidently describing objects or relationships that aren't in the image. To fix this, researchers typically use Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO).
However, the authors identify a major bottleneck: entropy-blindness. Current alignment methods treat every token in a response with equal weight. In a Chain-of-Thought (CoT) response, most tokens are filler, but a few "fork tokens" are critical—these are the branch points where the model commits to a specific visual claim (e.g., deciding if a shirt is "red" or "blue").
Furthermore, multimodal preference data is expensive and scarce. The authors show that under a small budget (3,000 prompts), standard pairwise objectives (DPO, mDPO) plateau at nearly identical performance levels. GEM-MM aims to break this plateau by focusing the learning signal on the model's internal uncertainty.
Methods
GEM-MM (Generative Entropy-guided Preference Modeling for Multimodal) shifts from offline pairwise learning to an on-policy loop consisting of three stages:
- Candidate Sampling: For a given image and prompt, the model generates $k$ different Chain-of-Thought completions.
- Cognitive Filtering (The Entropy Reward): The response is split into a "reasoning prefix" and a "final answer."
- Fork Reward: The model is rewarded for having high entropy (uncertainty/exploration) at its top "fork" tokens during reasoning.
- Final Penalty: The model is penalized for high entropy in the final answer, encouraging decisiveness.
- SEGA (Group-Normalized Update): Instead of using an absolute reward, the authors use a group-normalized policy gradient. This compares the $k$ candidates against each other, upweighting the ones that are more "fork-heavy" and "answer-decisive" than their siblings.
This approach allows the model to learn from its own internal consistency (or lack thereof) rather than relying solely on static, pre-annotated pairs.
Datasets
- Training: The model was trained on a "locked" budget of 3,000 prompts from the MM-RLHF dataset.
- Evaluation:
- AMBER: A benchmark for generative hallucination (metrics include CHAIR, Hal, Cover, and Cog).
- HallusionBench: Evaluates entangled language hallucination and visual illusion across three granularities.
- POPE & MMHal-Bench: Used for additional robustness checks.
- External Reward Models: The authors used Skywork-VL Reward and IXC-2.5-Reward as objective-agnostic "judges" to validate their results.
Key Insights and Ideas
- The "Fork" Matters: The most surprising finding is that rewarding uncertainty during the reasoning phase is the primary driver for reducing hallucinations. When the "fork" term was removed in ablation studies, the hallucination rate jumped back up to baseline levels.
- Saturation of Standard Methods: The paper quantifies "entropy-blind saturation," showing that on small budgets, it almost doesn't matter how you construct your preference pairs if you're using standard DPO; they all land within a tiny margin of each other.
- Style vs. Substance: The authors candidly note that entropy-based rewards can be "hacked" by the model changing its style (e.g., becoming more terse). They emphasize that while their "entropy-depth" metric tracks optimization, the real proof of success lies in the behavioral benchmarks like AMBER.
- The Collapse Mode: If trained too long (past the 3,000-prompt budget), the model enters a "collapse mode" where it learns to emit high-entropy gibberish to maximize reward while completely abandoning the task format.
Key Results
GEM-MM outperformed all preference-tuned baselines (MM-DPO, mDPO, SIMA) on a Qwen3-VL-4B-Instruct backbone:
- Hallucination Reduction: Reduced AMBER generative hallucination from ~24.0 to 20.6 and CHAIR scores from ~5.1 to 4.6.
- Preference Accuracy: Achieved the highest objective-agnostic implicit-reward accuracy (41.4%) and A/B choice accuracy (68.3%).
- Benchmark Sweep: Ranked first across all three granularities of HallusionBench.
- External Validation: The Skywork-VL Reward model, which never saw the GEM-MM objective during training, ranked GEM-MM as the top-performing system among those compared.
Limitations and Future Work
- Model Scale: The study was limited to a 4B parameter backbone. It remains to be seen how these entropy signals scale to 70B+ models.
- Format Compliance: The primary GEM-MM configuration saw a slight drop in answer-format compliance (66.5% vs 72.4% for SFT). While a "conservative" operating point fixed this, it sacrificed some of the entropy-depth gains.
- Refusal Behavior: The authors noted that gains in hallucination reduction did not automatically translate to better "safety" or refusal behavior, suggesting safety alignment requires separate handling.
- Future Direction: The authors suggest that "quality gates" (inspecting for repetition or loops) should be standard practice when using entropy-based rewards to prevent the documented collapse modes.
Summary updated