Does the Image Pick the Entity? Tracing Popularity Overshadowing Through Multimodal Reasoning Chains

When a question says "Jordan" or "Paris" and the accompanying image shows a small town in Texas or a computer science professor, a vision-language model has two sources of evidence about which entity is meant. One is the picture. The other is its memory of which "Paris" is usually meant, a popularity prior learned from pretraining text. Text-only entity linking has shown that the prior often wins. On ShadowLink, linkers pick the same entity whether or not the context points elsewhere (Provatorova et al.). Text-only language models obey a log-linear law: hallucination grows with the popularity ratio, with context length, and, perversely, with model size (Zhang et al.). Separately, multimodal work shows that long chain-of-thought (step-by-step written reasoning) drifts away from the image (Luo et al.). Nobody has asked what happens when these meet: a multi-step visual question whose answer depends on first resolving an ambiguous entity correctly.

This note makes a specific, testable claim. In multimodal reasoning, popularity overshadowing is a commitment event: the reasoning chain names an entity at some step, and from then on it reasons about that entity. Whether the image controls that commitment can be measured directly by swapping the image and checking whether the committed entity changes. We predict that the image's control over the commitment weakens the later the commitment happens in the chain and the larger the popularity gap between the two candidates. If so, step-by-step prompting should hurt most precisely on less-famous "shadow" entities. It also suggests a cheap fix: intervene once, at the commitment span, rather than at every generated token.

Core Claim and What Is New

The closest existing work is Generative Multimodal Entity Linking (GEMEL). It maps image features into a frozen language model's input and decodes an entity title constrained to knowledge-base names. On WikiDiverse it raised rare-entity accuracy from 37.5% (text only) to 57.5% (with the image). GEMEL shows the image helps when the entity is the whole answer and is produced in one step. It never examines a reasoning chain, so it cannot tell us whether that visual benefit survives when the entity is named mid-reasoning and then used for further deduction.

Three contrastive-decoding methods are the closest in mechanism. CoDA and self-contrastive decoding (Zhang et al., Zhang et al.) contrast the full prompt against a masked prompt in text-only models. Visual Contrastive Decoding (Leng et al.) subtracts the predictions made from a noise-corrupted image at every token. Rationale-Enhanced Decoding (Yamaguchi et al.) multiplies image-conditioned and rationale-conditioned predictions, assuming the rationale itself is sound.

Our proposal differs from all three in two ways. First, its measurement unit is the entity commitment inside the chain, not the token or the final answer. Second, the reference it contrasts against is a neutral image: a multimodal analogue of ShadowLink's context-free probe ("I saw [mention] today"). A noisy image amplifies generic object co-occurrence priors; a neutral image instead isolates the model's prior over entities. The diagnostic study is the primary contribution. The intervention is a secondary test of whether the diagnosis is actionable.

Research Program

Benchmark construction. We build a benchmark of entity-reasoning items from name-twin pairs. These pairs come from AmbER (Chen et al.) and ShadowLink, with images and entity grounding drawn from WikiDiverse and WikiMEL. Each item has three parts: an image whose content identifies one twin, a mention that is ambiguous on its own, and a question needing one to four reasoning hops over that entity's knowledge-base facts. A one-hop example is "the birth year of the person shown." A three-hop example is "years between their doctorate and the founding of their employer, given the chart of enrollment."

A chart subset follows Chart101 and ChartQA (Razeghi et al., Masry et al.). In these items the plotted data belong to the shadow entity, and titles are removed so the title cannot leak the answer, a shortcut Razeghi et al. documented. Every item exists in three versions: the shadow-identifying image, the head-identifying image, and a neutral image.

Popularity ratio is measured from Wikipedia pageviews between the twins. Each item also gets a direct-disambiguation "easy" twin question that asks only which entity is meant. This pairing lets us compute a hard-to-easy consistency score of the kind used in ConsisEval (Yang et al.).

Measurement. For each generated chain, an automatic linker maps the first entity named to a knowledge-base entry; we audit a sample by hand. Each chain then yields three quantities. The first is the commitment position: how many tokens into the chain the entity is first named. The second is the visual override rate: the fraction of items where swapping the shadow image for the head image changes the committed entity. The third is prior agreement: the fraction where the committed entity under the real image matches the one under the neutral image.

We then fit hallucination against the logarithms of popularity ratio, commitment position, and model scale. This tests whether Zhang et al.'s law carries over to the multimodal setting, with commitment position standing in for context length. We also test whether the dependence on scale keeps its inverse sign. Mixed Signals (Pezeshkpour et al.) found that models lean more on the image as tasks grow harder, which predicts the opposite of drift. Our hop-depth axis pits these two predictions against each other directly.

Intervention. The intervention, commitment-gated contrast, runs only when the chain is about to name an entity. At that span it compares the model's predictions under the real image with those under the neutral image and boosts candidates the image supports beyond the prior. It then inserts the chosen entity's short description into the chain, borrowing the re-anchoring idea from VERBALIZED (Rücker and Akbik), and lets reasoning continue normally.

Running the intervention once per item, not once per token, should keep cost close to plain decoding. The doubled per-token cost of VCD and Rationale-Enhanced Decoding is a known drawback of those methods.

Models are open-weight systems where we can read the model's output scores: LLaVA-1.5 at 7B and 13B, Gemma-3-4B, and Qwen2.5-VL at 7B and 72B. This span gives us a scale axis.

Baselines

The naive reference points are direct answering without step-by-step reasoning and standard chain-of-thought. Both are run on identical items with the same sampling settings. These two arms test the central prediction that chain-of-thought hurts more on shadow entities than on head entities. Beating them means higher shadow-tier accuracy without losing head-tier accuracy.

The strongest single-step entity baseline is GEMEL, retrained on our training split. It reports 86.3% accuracy on WikiDiverse and 82.6% on WikiMEL, with 57.5% on WikiDiverse tail entities (GEMEL). We use it to measure the easy-question disambiguation ceiling. Our method should match it on the easy questions and keep that accuracy through multi-hop chains, which GEMEL does not attempt.

The strongest decoding baselines are VCD, CoDA adapted by masking the image instead of a text constraint, and Rationale-Enhanced Decoding. Each is tuned on the same development split with the same hyperparameter-search budget, and wall-clock time is reported alongside accuracy. The only numbers the file gives for these methods come from other tasks: a POPE F1 gain of up to +7.4 for VCD, +27.9% on the Overshadow dataset for CoDA, and +5.87% on GQA for Rationale-Enhanced Decoding. No reported result was found on Lacuna for any method on multi-hop multimodal reasoning over ambiguous entities, so all comparisons will be run fresh.

The key ablation swaps the neutral-image reference for a noise-corrupted one while keeping the commitment gate. This isolates whether the entity-prior reference is the active ingredient. A second ablation applies the neutral-image contrast at every token instead of only at the commitment. It tests whether localizing the intervention matters for accuracy or only for cost.

Why not just…?

Why not just link the entity first with a strong linker and paste it into the prompt? This is the most serious threat, and we include it as a full baseline. VERBALIZED reaches 66.3% F1 on the ShadowLink shadow subset, and GEMEL could serve as the multimodal linker. The concern is that a pasted entity name is itself a context fact competing against a strong parametric prior. Zhang et al.'s NQ-Swap and Overshadow results show models override exactly such facts. We will measure how often a correctly pasted shadow entity is replaced by the head entity later in the chain. If link-first matches commitment-gated contrast, we will report that, and the contribution narrows to the diagnosis plus the recommendation to link first. That would still be a useful finding.

Why not just run Visual Contrastive Decoding? VCD's noisy image amplifies generic object co-occurrence, not knowledge-base entity priors, and it was evaluated only on object existence and counting (Leng et al.). The noise-versus-neutral ablation answers this question directly. If the two references perform the same, the entity-prior framing adds nothing.

Why not just transcribe the image to text first? IsoBench's IsoScratchPad improved science problems by up to 14.4 points (IsoBench). However, a photo of a professor rarely transcribes into text that names the professor. Longer context also raises overshadowing under the log-linear law. We include a caption-then-reason arm to test this.

Why not just use a larger model? Zhang et al. predict that larger models overshadow more. Our scale axis tests whether that prediction holds once an image is present.

Decisive Checks

The diagnostic claim fails if the visual override rate does not fall as commitment position grows. It also fails if the chain-of-thought penalty is the same for head and shadow tiers.

The intervention matters only if it beats both link-first and token-wise VCD on shadow-tier multi-hop accuracy at comparable cost. Its gains must also shrink when the neutral reference is replaced by noise. As a further check, the hard-to-easy consistency score should rise, which would mean the method fixes disambiguation instead of letting models answer multi-hop questions through shortcuts.