Vision-language models (VLMs), which answer questions about images, often describe objects that are not there, accept false premises, and say "yes" to almost any "is there an X?" question. The current fixes are costly. One option is to retrain the model with reinforcement learning on preference data, which requires multiple model copies in memory and sampling in the training loop. The other is to change decoding so that every token pays extra compute. Both treat every query the same. This note proposes a cheaper alternative. A small decision model reads the frozen VLM's internal state after a single forward pass over the image and question, before any answer is generated. It then picks one of three actions: answer directly, first fetch verified external evidence about the image and answer conditioned on it, or abstain.
The core bet is that the signal for "this answer will be hallucinated" is clearer in the model's middle-layer representations than in its output probabilities. Prior work found two relevant facts. Representation quality drops sharply where image features are projected into the language model, and hallucinated and faithful states are linearly separable in middle layers. Output confidence, in contrast, is badly miscalibrated exactly where hallucination comes from language priors. If the bet holds, most of the hallucination reduction that expensive alignment buys could be recovered with a router trained on logged outcomes, at a small fraction of the training cost and with nearly no inference overhead.
Core Claim
The claim has three parts, and each can be tested. First, a lightweight classifier on middle-layer and projector-boundary hidden states of a frozen VLM predicts, before decoding, which of three actions will produce a correct, non-hallucinated answer. It does this better than routers that use the model's own output confidence. Second, because the action set is small and each query is a one-step decision, the router can be trained from fully observed outcomes. We run every action once per training query and fit a cost-sensitive classifier. No policy rollouts, value network, or reward model sit in the loop, so the reinforcement learning compute bottleneck disappears instead of being optimized. Third, combining the router with a retrieve-and-refine evidence path gives hallucination rates competitive with weight-updating alignment methods, while calling the evidence path on a minority of queries.
Grounding in Prior Work
Two papers motivate the representation-level signal. Understanding Multimodal Hallucination with Parameter-Free Representation Alignment introduces Pfram, a training-free measure of how well a model's internal image representations agree with ground-truth object labels. It shows a sharp drop at the vision-to-language projection layer and a correlation of up to 77.62% with POPE performance. It also reports that LLM parameter count and projector type correlate much more weakly with hallucination than Pfram does. Pfram, however, is a model-level diagnostic. It needs ground-truth labels and cannot score an individual query at test time, so we cannot route on it directly. SHARP supplies the per-query version of the evidence: hallucinations driven by text priors and by visual-context conflicts are linearly separable in layers 10–15 of LVLMs. SHARP uses this to add fixed steering vectors to every query. We use the same separability only to decide what to do, and we leave the representation untouched. That matters because SHARP's steering cannot introduce facts the model lacks, whereas an evidence path can.
The evidence path reuses the retrieve-and-refine method of Retrieve and Refine. In that method, a generator conditions on a retrieved, human-verified prototype and learns to ignore it when it is irrelevant. We also draw on the hybrid-arbitration designs of An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems and A randomized prospective study of a hybrid rule- and data-driven virtual patient. Both use a meta-classifier to arbitrate between a high-precision verified path and a flexible neural path. Those arbiters read text and dialogue state. Ours reads the VLM's internal image-text representation, which is where the failure actually originates. ECVL-ROUTER is the closest routing system. It sends queries between small edge and large cloud VLMs using external text and image embeddings, at about 0.016 s overhead. It routes between model sizes rather than between generation, evidence lookup, and refusal, and it does not target hallucination metrics. We adopt its overhead budget as a constraint.
Three papers constrain the design or serve as baselines. GenAlign and RLHF-V are the alignment methods we must match or beat. The first uses GRPO training with a generative reward model. The second uses dense DPO on 1,400 segment-level human corrections. The Hallucination Tax of Reinforcement Finetuning shows that reinforcement finetuning cuts refusal on unanswerable questions by over 80%. This is why abstention is an explicit action: a router that only chooses between answer modes would inherit the tax. Generate, but Verify (REVERSE) detects uncertainty during decoding and backtracks. That costs a variable number of extra decoding passes and requires fine-tuning on 1.3 million samples. Our decision is made once, before decoding.
Research Program
The base models are LLaVA-v1.5 and LLaVA-Next, both frozen; Qwen-VL is a third check on generality. For each training query, we cache several representations from one forward pass over image and question. These are the projector output for the image tokens, the hidden states at the last question token in layers 8–16, and, for comparison only, the final next-token distribution.
The evidence path is deliberately non-parametric and verifiable. Image regions are matched by embedding similarity against a store of human-annotated regions built from OpenImages V7. The labels and short descriptions attached to the matched regions are passed to the VLM as an evidence block, in the retrieve-and-refine style. A detector-produced object inventory is an ablated alternative to this store. The abstain action emits a templated refusal or premise correction.
The router is a two-layer network of a few million parameters. Training data come from the HaloQuest training split and from POPE-style existence, attribute, and relation questions generated from OpenImages V7 annotations, disjoint from the COCO evaluation images. We execute all three actions once per training query and score each outcome. POPE-style items are scored exactly. HaloQuest items are scored with its Gemini auto-evaluator, which HaloQuest reports at $r=0.99$ with humans. The router then chooses
$$\pi(h) = \arg\max_{a} \big[\hat r(h,a) - \lambda\, c(a)\big],$$
where $h$ is the cached hidden-state feature, $a$ ranges over the three actions, $\hat r(h,a)$ is the router's predicted probability that action $a$ yields a correct answer, $c(a)$ is that action's measured wall-clock cost, and $\lambda$ is a price on compute. Sweeping $\lambda$ traces an accuracy-versus-cost frontier. This is plain cost-sensitive classification from fully observed outcomes. Calling it reinforcement learning would add nothing, and the absence of rollouts is the point.
The full training cost is three generations per training query plus fitting a small network. We will report it in GPU-hours alongside the costs of GenAlign and RLHF-V. Two analyses accompany the main results. A layer sweep tests whether routing accuracy peaks at the projector boundary and middle layers, as Pfram and SHARP predict. A per-category breakdown on HaloQuest covers false premise, insufficient context, and visual complexity, and checks whether each category is routed to the action we would expect.
Baselines
The naive reference point is the frozen model answering every query directly. Its reported numbers include LLaVA-1.5 at 14.52% on HaloQuest, as reported in GenAlign, and LLaVA-v1.5 at CHAIRs 52.8, as reported in SHARP. A second naive reference is "always ground", which applies the evidence path to every query. It sets the accuracy ceiling for the evidence path and the cost the router must undercut.
The strongest existing method on the question of matching alignment-level hallucination reduction cheaply is GenAlign. It raises LLaVA-1.5 on HaloQuest from 14.52% to 45.16%, the largest reported gain in the file on our main false-premise benchmark. We will use released GenAlign checkpoints where available. Otherwise we reproduce it on the same LLaVA-1.5 base with the same HaloQuest training split our router sees, and log its GPU-hours. Beating it means reaching at least 45.16% HaloQuest accuracy on LLaVA-1.5 with at most one tenth of GenAlign's training compute. Matching it within two points at that compute ratio would still be a meaningful efficiency result.
On open-ended captioning and POPE, the strongest inference-time competitor is SHARP. It reports CHAIRs 34.8 on LLaVA-v1.5, 82.8% POPE adversarial accuracy on LLaVA-Next versus 79.5% for standard sampling, and 7.04 s per sample versus 13.97 s for VCD. We run it with its published layer-10 injection on the same models and prompts. Beating it means lower CHAIRs and higher POPE adversarial accuracy at no greater mean latency. RLHF-V (a 34.8% object-hallucination reduction on Muffin, under one hour on 8 A100s) and REVERSE (a 12% MSCOCO hallucination reduction and a 34% HaloQuest improvement) are run on the same bases where checkpoints allow. They are compared on both accuracy and per-sample latency variance.
Two ablations isolate the key ingredient, which is reading internal representations. The first is an output-confidence router: the same router architecture and training data, but its only input is the final next-token distribution plus first-token entropy. The second is a router given only external image and text embeddings, as in ECVL-ROUTER. All routers get the same tuning budget, a fixed number of hyperparameter trials, and are compared at matched evidence-call rates so that differences cannot come from simply grounding more often.
Why not just…?
Why not just always retrieve evidence and skip the router? This is the most serious threat, and the proposal is built so it could lose to it. If always-ground matches the router's accuracy at acceptable cost, the router is unnecessary. Two things could still favor the router. Evidence costs latency on every query, and injected text can strengthen text-prior hallucination, which SHARP identifies as one of the two main causes. The experiment that settles it compares router and always-ground on POPE, AMBER, HaloQuest, and MME. The router must match always-ground's hallucination metrics while calling evidence on well under half of queries, and it must avoid always-ground's losses on MME perception and AMBER attribute and relation questions. If it does neither, we report that always-ground suffices.
Why not just route on the model's output confidence? Output confidence is free and is what most hybrid arbiters use. However, Evaluating Object Hallucination in Large Vision-Language Models documents "yes" rates near 99% under some prompts, so confident output is exactly the failure mode. The output-confidence ablation tests this directly. If it comes within one point of the hidden-state router on POPE adversarial and HaloQuest false-premise at matched call rates, our central claim fails and we will say so.
Why not just run full reinforcement learning alignment, as GenAlign or RLHF-V do? These methods work, and GenAlign's 45.16% is the bar. But they update weights with rollouts in the loop and, for RLHF-V, need human segment corrections. The hallucination-tax result also warns that reward-driven finetuning can erode abstention. The compute ledger and the HaloQuest insufficient-context breakdown show whether a frozen model with a router reaches comparable accuracy more cheaply. Applying our router on top of a GenAlign checkpoint tests whether the two are complementary.
Why not just apply SHARP's steering everywhere? Steering is cheap and effective, but it uses one fixed vector for all queries and cannot add missing facts. The head-to-head on CHAIR and POPE settles this. So does a combined condition in which the router chooses between steered-direct and grounded answering.
Proposed Experiments and Decisive Checks
The evaluation suite is POPE across its random, popular, and adversarial splits; CHAIR on MSCOCO captions; AMBER for existence, attribute, and relation hallucination; and the HaloQuest test split broken down by category. MME perception serves as a capability-preservation check, with SHARP's reported 510.0 to 613.3 on LLaVA-v1.5 as context. Every condition also reports mean and 95th-percentile latency per sample, the fraction of queries sent to each action, and training GPU-hours.
Three results decide the proposal. The first is whether the middle-layer router beats the output-confidence router at matched evidence-call rates on POPE adversarial and HaloQuest false-premise. If not, the representation-level claim is wrong. The second is whether the router on a frozen LLaVA-1.5 reaches GenAlign's 45.16% HaloQuest accuracy within a tenth of its compute. If not, the cost argument is weak. The third is whether the router matches always-ground's hallucination rates with far fewer evidence calls and without its capability losses. If not, a fixed pipeline suffices.
A further robustness check deliberately corrupts the evidence store by swapping region labels at increasing rates. This tests whether the router learns to stop calling evidence as it becomes unreliable. A static pipeline cannot do this, and it is the property that justifies a learned arbiter.
References
- Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Retrieve and Refine: Improved Sequence Generation Models For Dialogue
- An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems
- A randomized prospective study of a hybrid rule- and data-driven virtual patient
- ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models
- GenAlign: Towards Unified Alignment Framework of MLLMs via Generative Reward Model
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback
- The Hallucination Tax of Reinforcement Finetuning
- Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
- HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
- Evaluating Object Hallucination in Large Vision-Language Models