Does the Connector Matter? A Tabular Benchmark for Hallucination in Vision-Language Models

Vision-language models sometimes describe objects that are not in the image. This failure is called hallucination. These models are built from a frozen image encoder, a language model, and a small trainable "connector" that turns image features into tokens the language model can read. Recent papers credit redesigned connectors with real hallucination reductions. Examples include feeding in features from many encoder layers, or sending each image token to specialized expert sub-networks. But each paper changes the training data, the language model, and the recipe at the same time as the connector. Neural architecture search went through the same situation. There, careful re-evaluation showed that training tricks moved accuracy by more than three points, while search algorithms beat random sampling by well under one (NAS evaluation is frustratingly hard).

We propose to do for connectors what tabular benchmarks did for image-classification cells. We will fix the recipe, exhaustively train a closed space of connector designs with repeated seeds, and publish every model's hallucination scores. The goal is to answer three questions with numbers. How much of the variation in hallucination comes from the connector, as opposed to seed noise and training recipe? Do connector rankings survive a change of image source and of negative-sampling difficulty? Do test-time anti-hallucination fixes stay useful on top of a good connector, or does a good connector make them unnecessary?

Core Claim

Our hypothesis has a deliberately falsifiable form. Once the recipe is held fixed, most connector choices will barely change object hallucination, staying within about one seed standard deviation. A single axis, namely which encoder layers the connector reads from, will account for most of the connector-attributable variance in the hardest probing setting. Rankings learned on COCO-style images will transfer poorly to other image sources and to adversarial negatives.

If this is right, much of the connector literature's hallucination gains are recipe effects. It would also mean that "search for a better connector" is the wrong research direction, and that "find which input the connector should see" is the right one. If it is wrong, and connector topology produces large, domain-stable effects, the benchmark becomes the first reproducible testbed for connector search. Either outcome is informative. The contribution is the controlled measurement, not a new search algorithm.

Research Program

Connector designs. The search space draws its axes from the two connector papers we want to disentangle.

  • Dense Connector for MLLMs compared three hand-built ways of combining multi-layer encoder features. These are concatenating tokens, concatenating channels, and summing within layer groups before concatenating. It also tested a 4x token downsampling from 576 to 144 tokens. It reports gains such as +2.7% on ScienceQA, but under its own data mixture.
  • MoCHA replaced the standard two-layer projector with a mixture-of-experts connector that routes each token to 2 of several small networks. It reports beating a 13B model by 0.85% on POPE with a 2.7B Phi-2 backbone, but also with a different backbone set and data mixture.

Neither paper searched systematically. Neither repeated seeds. Neither tested whether its choice transfers across image domains.

From these, our space has five axes:

  • Encoder layers read: final layer only, final plus one mid-layer group, final plus two groups, or all groups.
  • Fusion operator: token concatenation, channel concatenation, or group sum.
  • Projector depth: one, two, or three layers.
  • Visual token count: 576 or 144.
  • Number of experts: one or four, with top-2 routing.

This gives 144 configurations. Each is trained with three seeds under one fixed recipe. The recipe uses the LLaVA-1.5-style two-stage schedule (connector alignment, then instruction tuning) on a fixed instruction subset, with a Phi-2-scale language model so the full grid is affordable.

Recipe and data variation. A random subset of 24 configurations is retrained under two further recipes:

  • a stronger recipe with longer training and the larger instruction mixture;
  • a data-rebalanced recipe that downweights frequent object co-occurrences.

This gives a direct estimate of recipe and data variance on the same configurations, which is the comparison that NAS evaluation is frustratingly hard showed is usually missing.

Recorded metrics. For every trained model the table records the following:

  • POPE (Evaluating Object Hallucination in Large Vision-Language Models): binary "is there an X in the image?" probing. We record F1 and yes-rate under random, popular, and adversarial negatives, on each of three image sources: MSCOCO, A-OKVQA, and GQA.
  • CHAIR: the fraction of hallucinated objects in free captions, at both object and sentence level.
  • MME perception subscores.
  • GQA accuracy, as a general-capability control.
  • Latency.
  • A probe score. Following SHARP, which found that faithful and hallucinated hidden states are linearly separable in middle language-model layers, we fit a linear probe at layers 10–15 on held-out HaloQuest-style stimuli. This tests whether better connectors shrink the representational conflict SHARP steers against.

Treatment of evaluation metrics. POPE is the primary metric because its prompt sensitivity is small (standard deviation 0.78 across templates, versus 3.22 for caption-based scores, per Li et al.). CHAIR is secondary.

Baselines

Naive reference. The do-nothing point is the standard LLaVA-style two-layer projector on final-layer features, trained under our fixed recipe with three seeds. Every connector effect is reported as a difference from this cell, in units of its seed standard deviation. A connector "beats" it only if its adversarial-POPE F1 advantage exceeds two seed standard deviations on all three image sources.

Strongest existing connector designs. These are Dense Connector's group-sum design and a MoCHA-style four-expert connector. Both are reimplemented as cells of our grid, so they share data, compute, and tuning budget exactly with every other cell. We will report how far the gains each paper claimed shrink under the shared recipe. The reported numbers we compare against are MoCHA's +0.85% POPE over SPHINX-Plus and Dense Connector's +2.7% ScienceQA, both from their respective papers.

Strongest existing hallucination fixes. These operate at decoding or representation level, without retraining:

  • RITUAL (RITUAL) feeds randomly flipped, rotated, or cropped copies of the image alongside the original during decoding. It reports raising LLaVA-1.5 POPE accuracy from 84.13% to 88.87%.
  • SHARP reports adversarial-POPE accuracy rising from 79.5% to 82.8% on LLaVA-Next, and CHAIR sentence-level hallucination falling from 52.8 to 34.8 on LLaVA-v1.5.
  • HALC (HALC) reports MiniGPT-4 CHAIR sentence hallucination dropping from 30.87% to 17.80%. The file gives no extracted POPE number for HALC. We run it from its released code (BillChan226/HALC) as a CHAIR baseline only.

We apply RITUAL and SHARP to the naive projector and to the best connector in the table, with identical decoding settings. The best connector "beats" a test-time fix if, at single-pass cost, it matches the fix's improvement over the naive projector.

Search baseline and key ablation. Random search over the table is the mandatory search baseline, because NAS-Bench-201 showed random search and regularized evolution outperforming gradient-based search. The ablation that isolates our central hypothesis freezes the layer-selection axis at "final only" and varies the other four axes. If connector-attributable variance collapses under this ablation, layer selection is the ingredient that matters.

Why not just…?

Why not just run a quick hand ablation, as Dense Connector did? A three-variant, single-seed ablation cannot separate a 0.5-point connector effect from seed noise or from the training recipe. In the DARTS space, 214 random architectures spanned only a 0.23 standard deviation, while recipe changes moved accuracy by more than 3 points (NAS evaluation is frustratingly hard). The repeated-seed grid and the recipe subset are exactly what such an ablation lacks.

Why not just use a decoding fix like RITUAL or steering like SHARP and leave the architecture alone? This is a real threat, and it may win on raw POPE. Those methods carry costs that matter in deployment:

  • HALC is 1.35x to 2.4x slower and depends on an external detector.
  • RITUAL doubles forward passes and warns that its transformations can hurt orientation-sensitive domains.
  • SHARP is cheap but must be fit to each model.

Our interaction experiment settles the question. If the best connector plus no fix roughly equals the naive projector plus RITUAL, architecture is a cheaper route. If RITUAL's gain is unchanged on top of every connector, the two are orthogonal, and connector work does not touch the mechanism these fixes target.

Why not just rebalance the training data, since hallucination tracks object co-occurrence? Li et al. attribute InstructBLIP's lower hallucination to its instruction data. Our rebalanced-data recipe on 24 configurations measures this directly. If data variance dwarfs connector variance, we will say so plainly; that would be a central finding, not a nuisance.

Why not just design a better search algorithm? In a well-constrained space, random search is usually within noise of sophisticated search. Proposing a new optimizer before knowing whether the landscape has meaningful variation would repeat NAS's history. The table makes any later search claim checkable in milliseconds.

Proposed Experiments and Decisive Checks

Variance decomposition. The first check splits the variance of adversarial-POPE F1, CHAIR sentence rate, and GQA accuracy into four parts: per connector axis, seed, recipe, and data. If seed noise rivals the effect of every axis except layer selection, the core hypothesis holds.

Ranking transfer across settings. The second check computes rank correlation of connector rankings in two directions:

  • from COCO POPE to A-OKVQA and GQA POPE;
  • from random negatives to adversarial negatives.

Li et al. report MiniGPT-4 falling from 78.86 to 71.37 F1 between random and adversarial settings, and NAS-Bench-201 shows CIFAR-10 winners not dominating ImageNet-16-120. We therefore expect weak transfer. A correlation below about 0.5 would mean connector papers tuned on COCO are partly fitting COCO's co-occurrence statistics.

Hallucination versus general capability. The third check asks whether hallucination rankings are just general-capability rankings. We regress POPE on GQA accuracy across the table and analyze the residual. A connector that only makes the model stronger overall is not a hallucination fix.

Probe agreement. The fourth check tests whether SHARP-style probe separability at layers 10–15 falls for connectors with lower adversarial hallucination. If it does, connectors are reducing the internal conflict between visual evidence and language priors, and are not merely shifting the yes-rate.

Release. The released artifact is the full table: 432 base runs plus recipe-variant runs, with training logs, checkpoints, and evaluation outputs. Future connector or search papers can then be evaluated without retraining.