In the quest to make Large Vision-Language Models (LVLMs) more reliable, researchers have turned to Preference Optimization (PO)—fine-tuning models on human or AI feedback to discourage "hallucinations" (mentioning objects not present in an image).
However, a new paper from Hansang Lee at Seoul Women’s University asks a provocative question: Are these models actually getting smarter, or are they just learning to shut up?
TL;DR
Preference Optimization (PO) reduces VLM hallucinations largely by making models stop writing sooner, which inadvertently kills object recall. By simply forcing the base model to stop early, researchers reproduced up to 86% of the hallucination reduction seen in state-of-the-art PO models, suggesting that current "alignment" might just be a fancy way of teaching models to say less.
What Problem Does This Paper Address?
Object hallucination is a major hurdle for LVLMs like LLaVA. While PO techniques (like DPO or RLAIF-V) successfully lower hallucination rates, they also tend to produce shorter, less descriptive captions.
The author investigates whether the "gains" reported by PO models are a genuine improvement in visual understanding or merely a side effect of reduced output length. If a model mentions fewer objects, it has fewer opportunities to be wrong. This creates a "brevity trap" where metrics like the hallucination rate improve, but the model's utility (its ability to describe everything in the image) declines.
Methods
The paper introduces a recall-controlled evaluation framework. Instead of comparing models at their default settings, the author compares them at the same level of object recall.
- EOS Logit Bias: To simulate "early stopping" in the base model, the author adds a positive bias to the end-of-sequence (EOS) token during decoding. This allows the base model to reach the same (lower) recall levels as the PO-tuned models.
- Negative EOS Bias Probe: Conversely, the author applies a negative bias to PO models to force them to write longer, testing if they can "recover" the objects they stopped mentioning.
- Base-Guided Object-Category Restoration: The author proposes a training-free intervention. It takes the output of a PO model and appends high-confidence object categories that were mentioned by the base model but missed by the PO model.
Datasets
The study evaluates four public PO checkpoints of LLaVA-1.5-7B (RLAIF-V, CHAIR-DPO, CSR, and HA-DPO) using:
- CHAIR (Caption Hallucination Assessment with Image Relevance): Specifically using 500 random COCO images (CHAIR-500) and a separate held-out set of 500 images.
- AMBER: A multi-dimensional hallucination benchmark.
- POPE: A benchmark for yes/no object existence questions.
- MMStar: To ensure general reasoning abilities aren't degraded.
Key Insights and Ideas
- The "Say Less" Strategy: PO models name significantly fewer object categories (2.1–2.8 per caption) compared to the base model (3.2). Surprisingly, simple early stopping of the base model reproduces 58% to 86% of the hallucination reduction (CHAIRi gain) of three major PO checkpoints.
- Lost Recall is Hard to Recover: When PO models are forced to write longer (via negative EOS bias), they don't start naming the missing objects. Instead, 94% to 96% of the additional text just repeats object categories they already named.
- The Base Model Still "Sees" More: Even when a PO model misses an object, the base model’s caption for the same image often includes it. This suggests the information is present in the model's internal representation but is being filtered out by the PO tuning.
- F1 Stagnation: While hallucination rates drop, the set-level object F1 score (which balances precision and recall) shows no significant gain for any PO model.
Key Results
- Early Stopping Parity: On COCO, the base model with a positive EOS bias matched the hallucination performance of RLAIF-V and CHAIR-DPO while maintaining similar recall.
- Restoration Success: The "Base-guided restoration" method brought three PO checkpoints back to within one point of the base model's recall. Crucially, it did this while maintaining a lower hallucination rate than the base model itself.
- Quantitative Gains: For RLAIF-V, restoration lowered the distinct-object hallucination rate ($H_u$) by 5.24 points compared to the base model at the same recall level.
Limitations and Future Work
- Scope: The study focuses on LLaVA-1.5-7B and greedy decoding. Results might differ for larger models or different architectures.
- Attribute/Relation Hallucinations: The paper focuses on object presence. Hallucinations regarding colors, counts, or spatial relations were not the primary focus.
- Fluency: The restoration method uses a simple template ("There is also a {object} in the image"), which may not be as fluent as a naturally generated caption.
- Future Training: The author suggests that future PO training should specifically target "grounded category coverage" to ensure that reducing hallucinations doesn't come at the cost of descriptive richness.
Summary updated