Orphaned Shadows: Separating Physical Evidence from Language Priors in Vision-Language Hallucination

When a vision-language model is shown a photo of a street with a bicycle digitally erased and still says "there is a bicycle," the usual diagnosis is that the language model guessed from context: streets usually have bicycles. That diagnosis may be wrong in a measurable fraction of cases. Erasing an object with an inpainting model (a generator that fills a masked region with plausible pixels) usually leaves its cast shadow, its reflection, and the dents and occlusions it made in the scene. The model may be reading real physical evidence that something stands just out of view or was there a moment ago, not inventing from word statistics. This proposal builds a controlled image set in which the object and its lighting traces are removed separately. It then measures how much of what benchmarks call "hidden hallucination" is driven by leftover illumination evidence and how much by language priors.

The illumination-forensics literature supplies the tool that makes the separation testable. That literature checks whether shadows, shading, and highlights in an image obey the geometry of a single light source. Using it, we can plant shadows whose direction either agrees or conflicts with the scene's light, and ask whether vision-language models treat a shadow as physics ("something casts this") or as texture ("shadows go with bicycles"). The outcome matters in practice. Current mitigation methods are tuned to suppress prior-driven guesses. If a sizable share of benchmark errors is evidence-driven, those methods are being scored partly on the wrong failure, and some of them, especially zoom-in methods, should make that failure worse.

Core Claim

The claim is that object-removal hallucination in large vision-language models (LVLMs, systems that pair an image encoder with a large language model) has at least two separable causes. The first is co-occurrence priors. The second is residual physical evidence, chiefly cast shadows. The two can be pulled apart with a factorial design in which object pixels and their shadow pixels are removed independently. A third question is whether models are sensitive to the physical consistency of a shadow, meaning whether its direction agrees with the scene's single light source, or only to its presence. That question can be answered by planting casterless shadows that are consistent or inconsistent with that light direction.

The motivating evidence comes from Behind the Magic, MERLIM. MERLIM inpaints objects out of images and finds that models keep asserting them, with roughly an 8% precision drop. It attributes this to grounding failure, but it does not check what the inpainter left behind. Evaluating Object Hallucination in Large Vision-Language Models (POPE) supplies the competing explanation: co-occurrence statistics predict which absent objects get claimed. Neither paper controls for physical traces.

The only paper in the file that joins illumination and vision-language models is When Lighting Deceives (ITA). ITA optimizes global relighting to break classification, captioning, and question answering. Our work differs from ITA in three ways. We use localized, forensically verified shadow edits rather than adversarial global relighting. We ask a causal attribution question rather than measuring an attack success rate. And we evaluate on hallucination benchmarks, which ITA does not touch.

Research Program

The core artifact is a factorial edit set built on sunlit MSCOCO validation images. These are the same image source as POPE, MERLIM, and ITA. We select scenes with a hard directional light and at least two clearly visible object–shadow pairs, and hand-annotate object masks and shadow masks for about 1,000 target objects.

From each target we generate five conditions:

  1. Original. The image is unedited.
  2. Object removed, shadow kept. This reproduces the MERLIM protocol.
  3. Object and shadow removed. Both masks are inpainted.
  4. Sham edit. A same-area region elsewhere in the background is inpainted, which controls for generic inpainting artifacts.
  5. Shadow only. The object is removed but its shadow is restored, so the image contains a shadow with no visible caster.

A second set plants copies of real shadows into object-free regions. Each planted shadow is oriented one of two ways. In the consistent orientation it fits the scene's light direction. In the inconsistent orientation it is rotated so it violates that direction.

Physical validity is checked with the forensic tools surveyed in Visual Computing and Machine Learning Techniques for Digital Forensics. In particular, the wedge constraint traces rays from shadow points through object points toward the light, and in an authentic scene those rays from all object–shadow pairs must meet in a common region. We use it in two ways. First, we confirm that consistent planted shadows fall inside the light region estimated from the image's remaining pairs and that inconsistent ones fall outside it. Second, we confirm that removal conditions leave no detectable traces of the removed pair. As a second check we run Disharmony, a reverse-harmonization detector built to find lighting-blended edits. It reports ROC AUC of about 0.70 where generic forensic models score 0.52–0.63. We discard edits it flags strongly, so that models cannot succeed by spotting inpainting rather than reading content.

On every image we ask POPE-style presence questions about the target object and about a co-occurrence-matched distractor. We also collect free-form captions and score them with CHAIR, which counts objects mentioned in a caption that are not in the image. The target object's co-occurrence strength with the remaining scene is computed from COCO annotations, following POPE's adversarial sampling logic.

The analysis is a logistic regression of "object claimed" on four predictors: condition, residual shadow area, shadow consistency, and co-occurrence strength. It is fit per model. Its output is the share of removal hallucinations attributable to residual shadows rather than priors.

A secondary mechanistic step follows SHARP. SHARP showed that hallucination states are linearly separable from faithful ones in middle layers (10–15) and steered them by cause. We train linear probes to test whether shadow-driven and prior-driven false "yes" answers occupy different directions in those layers. If they do, we test whether SHARP's prior-steering vector suppresses the prior-driven errors while leaving the shadow-driven ones untouched.

Baselines

The naive reference point is the MERLIM protocol itself: object removed, shadow kept, with the resulting false-claim rate read as pure grounding failure. MERLIM reports an approximate 8% precision drop on inpainted images. Our result "beats" this reading if removing the shadow as well lowers false-claim rates by a statistically reliable margin that the sham edit does not reproduce. That would show the standard number mixes two causes.

The strongest competing explanation is the co-occurrence prior. We implement it as a predictor that uses only the COCO co-occurrence strength between the target and the remaining objects, the quantity POPE uses to build its adversarial split. It sees exactly the same images and questions. The physical account wins only if residual shadow area adds significant explanatory power beyond co-occurrence. We measure this as held-out log-likelihood gain under cross-validation, on the same image split for both predictors.

The strongest existing mitigation methods are RITUAL, HALC, and SHARP, each run from released code or settings on the same backbones with default hyperparameters and no tuning on our set.

  • RITUAL (RITUAL) combines predictions from the original image and randomly transformed copies. It reports LLaVA-1.5 POPE accuracy rising from 84.13% to 88.87%.
  • SHARP reports Adversarial POPE on LLaVA-Next rising from 79.5% to 82.8%.
  • HALC (HALC, run via billchan226/HALC) contrasts predictions on zoomed crops against the full image. It reports MiniGPT-4 CHAIR dropping from 30.87% to 17.80%.

No reported result exists on Lacuna for any of these methods on object-removal or shadow-edited images. We therefore measure all of them ourselves across the five conditions. The question is not whether we beat them but whether their gains concentrate in the prior-driven cells and vanish or reverse in the shadow-only cell.

The ablation that isolates the key ingredient, physical consistency, is the consistent-versus-inconsistent planted-shadow contrast. Shadow area, texture, and position are matched, and only direction relative to the scene light differs. If consistent shadows elicit more caster claims than inconsistent ones, the model is using lighting geometry. If they elicit equal rates, it treats shadows as texture.

Why not just…?

Why not just use a better inpainter? Stronger inpainters are trained to produce plausible scenes. A plausible scene often keeps the shadow, because the shadow belongs to the ground region the inpainter is told to preserve. Whether a given inpainter removes shadows is exactly what has to be measured. Our object-and-shadow condition is the "better inpainter" done deliberately and verified with the wedge-constraint check. If modern inpainters already remove shadows reliably, conditions 2 and 3 will look alike, the effect will be small, and that is itself a reportable negative result.

Why not just measure priors with POPE's adversarial split? POPE's adversarial split controls which absent object is queried. It does not control what physical evidence the image contains about that object, so it cannot separate the two causes. Its unedited images never contain an orphaned shadow at all. The regression above directly tests whether co-occurrence alone explains removal hallucinations. This is the real threat to the proposal: if co-occurrence absorbs nearly all the variance, the physical account fails and we will say so.

Why not just apply contrastive or zoom-in decoding and see if it fixes things? We do apply it, but we expect different methods to fail in different directions. HALC's pilot study found that for 84.5% of hallucinations some zoomed crop yields correct tokens. Its crops are located by an object detector, though, and fall back to unguided crops when no box is found. A crop that lands on an orphaned shadow could strengthen the shadow evidence rather than weaken it. Inharmonious Region Localization via Recurrent Self-Reasoning shows the opposite need: judging whether a region fits the scene's lighting requires comparing it against global background statistics. So we predict HALC's benefit shrinks in the shadow-only cell while RITUAL's global transformations are less affected. Observing that crossover would be direct evidence that local focusing and global lighting reasoning trade off.

Why not just ask the model whether there is a shadow? We will. A model that correctly reports "a shadow of a bicycle" while answering "no bicycle" is grounded. One that says "a bicycle" is over-inferring. That answer helps scoring, but it does not by itself show whether shadows cause the false object claims. Only the factorial removal design can show that.

Decisive Checks

We will run LLaVA-1.5, LLaVA-Next, InstructBLIP, MiniGPT-4, and mPLUG-Owl2. The study has three decisive outcomes.

The first is the yes-rate gap between "object removed, shadow kept" and "object and shadow removed," with the sham edit near the "original" absent-object baseline. A gap of several points, consistent across models, means MERLIM-style numbers conflate evidence with priors. A near-zero gap means priors dominate and the physical account is rejected.

The second is the consistent-versus-inconsistent shadow contrast. A positive difference would be the first evidence that LVLMs use lighting geometry when they infer objects. Given ITA's finding that relighting shifts these models substantially and the "independent eye" failure that Exposing GAN-generated Faces Using Inconsistent Corneal Specular Highlights documents for 2D generators, we expect weak or no sensitivity. That expected null is still informative, because it says the shadow effect is texture association rather than physical reasoning.

The third is whether the mitigation methods' gains are cause-specific, and whether shadow-driven and prior-driven errors are linearly separable in middle layers. Together these determine whether future hallucination benchmarks need a physical-evidence control.