A popular way to test whether a vision-language model (a language model that also reads images) actually looks at the picture is to erase an object and ask whether it is still there. If the model says "yes, there is a racket" after the racket has been painted out, we conclude it answered from habit, not from the pixels. Benchmarks built this way, such as ROHE and MERLIM, rest on an unchecked assumption: that the edited image is a faithful picture of the scene without the object. The editing tools that erase objects are themselves generative models with known limits. They leave smudges, ghost shapes, and leftover shadows. The source images are also public photos that the tested models have very likely seen during pretraining. So a "yes" on an erased image has at least three possible causes. The model may be guessing from scene context. It may be reacting to real traces the eraser left behind. Or it may remember the original photograph.
We propose to measure how much of removed-object hallucination comes from each cause. The tool is a new evaluation set in which the object is physically taken out of a real scene and the scene is photographed again from the same position. This yields the true "without" image, which the object-removal literature normally treats as unobtainable. Against this reference we can grade each automatic eraser by how closely it reproduces the model behavior seen on real absence. We can also check whether current hallucination fixes teach models to see absence or only to spot the signature of a particular eraser. Finally, we can extend the test from yes/no presence questions to multi-step questions in which the erased object is one premise of a longer chain of reasoning.
Core Claim
Our claim is that the measured failure rate of models on erased-object tests is a sum of three separable effects. The first is a scene-context prior. The second is an eraser-trace effect, which is bounded by the generative ceiling of whatever inpainter produced the edit. The third is a memorization effect that comes from benchmark images being part of pretraining. Current benchmarks report only the sum. We predict that the eraser-trace share is large for LaMa-style erasers and shrinks, without vanishing, for cleaner amodal completion. We predict that the memorization share is measurable on MSCOCO-derived items and absent on freshly captured scenes. We further predict that a mitigation trained on eraser-generated negatives can partly learn to detect eraser artifacts: its gains on inpainted test sets would then shrink on physically removed scenes. If all three predictions fail and model behavior on inpainted and physically removed images is indistinguishable, that is also a useful result. It would certify the existing benchmarks as valid.
Why This Gap Is Real
Two benchmarks motivate the study. ROHE removes objects from MSCOCO 2017 with LaMa and scores models with a strict paired accuracy called acc+, which requires a correct answer on both the original and the edited image. It then trains a preference-optimization fix, oDPO, on removal-generated negatives. MERLIM uses inpainting at a scale of 300,000 image-question pairs to expose "hidden hallucinations." Both treat the eraser as a black box and both draw on public images. We do not propose a better hallucination benchmark of the same kind. We audit what these benchmarks measure.
A third line of work constrains the design by showing that erasers are not neutral. EntityErasure shows that diffusion inpainters fill erased regions with unintended new objects, which it calls "sundries." It introduces reference-free counts of them: Mean Sundries Num (MSN) and Mean Area Ratio of Sundries. It also reports large differences between erasers. The object-removal literature explains why no one has checked edits against truth. Mori et al. note that ground truth of the scene without the object is generally unavailable, and Oh et al. show that full-reference metrics against the original image penalize successful removal. The field has therefore moved to reference-free scores: ReMOVE compares deep features of the filled patch with its surroundings, and the CFD score penalizes object-like hallucinated structure. Our paired captures do not replace these metrics. They let us test them, by asking which reference-free score best predicts whether a model will be fooled.
The contamination direction supplies the second confound. DyVal shows that text models with strong static-benchmark scores collapse on freshly generated problems, but its evidence is confined to text. CharXiv and VisionGraph supply the third ingredient. Both show that an early perceptual error propagates through otherwise valid reasoning. This motivates testing erased objects as premises in multi-step questions, not only as yes/no targets. CofactGVR is the closest method-side neighbor. It trains models with a reward gap between reasoning over a predicted evidence box and reasoning over a noise patch pasted in its place. It never asks whether the model distinguishes true absence from visible corruption, and that distinction is exactly what our design isolates.
Research Program
The central asset is a paired physical-removal set. We will capture roughly 1,000 indoor and tabletop scenes on a locked tripod with fixed exposure. In each scene, two to four movable objects are photographed in place and then physically removed one at a time. Each removal gives a real "without" image that includes the true disappearance of the object's shadows, reflections, and contact darkening. OmniPaint and EntityErasure identify these effects as the hardest for erasers. Small lighting drift between shots is the main risk. We will measure it on regions far from the object and discard pairs above a threshold.
Each "with" image is then erased by several automatic methods, using the same object mask:
- a solid or noise patch;
- LaMa, the ROHE eraser;
- PowerPaint and CLIPAway, diffusion erasers with high sundry counts;
- EntityErasure, the cleanest eraser reported.
This gives, for the same scene and object, one real absence and a spectrum of synthetic absences. The spread across erasers is the eraser-trace effect.
To measure memorization, we pair the fresh captures with a matched MSCOCO-derived set. This set uses the same object categories, similar scene types, and the same erasers and question templates. A gap in hidden-hallucination rate between COCO and fresh images, with eraser held fixed, estimates the memorization share. We add a direct recall probe. The model sees a heavily blurred COCO image and is asked to list objects, and we measure how often it names objects from the original COCO caption above a category-frequency baseline. The probe tells us whether a given COCO item is plausibly remembered, so hallucination can be stratified by recall.
To measure the scene prior, we take the residual failure rate on physically removed fresh scenes. No eraser traces or memory remain there, so what is left is the model's own expectation given context. We also vary how strongly the scene implies the object. A racket is strongly implied on a court, while a mug is weakly implied on a desk.
Finally, we extend beyond presence questions. For every pair we generate templated multi-step questions whose answer depends on the removed object. These include counts ("how many chairs"), spatial chains ("what is directly left of the object left of the lamp"), and conditional questions whose correct answer changes when the object is absent. Because the scenes are new captures and the questions are generated from per-scene annotations, the question set can be regenerated at will, in the spirit of DyVal. This lets us ask whether an eraser-induced presence error cascades into the downstream answer. It also lets us ask whether step-by-step prompting amplifies the cascade, as VCBench reports for perception-heavy tasks.
Baselines
The naive reference point is the existing ROHE protocol taken at face value: LaMa erasure on MSCOCO, scored with acc+. Here the reported LLaVA-1.5-7B baseline is 39.21% acc+, rising to 61.65% with oDPO (ROHE). We will re-run this exact protocol with the same models and the same prompt. The aim is to reproduce these numbers on our hardware before any comparison.
The strongest competing counterfactual generator is EntityErasure. It reports an MSN of 0.122 on EntitySeg, against 0.554 for PowerPaint and 0.679 for CLIPAway (EntityErasure). If any automatic protocol can already stand in for real absence, it is this one. For each eraser, the comparison metric is agreement between model verdicts on the erased image and on the physically removed image of the same scene, pooled over models. Fairness comes from sharing masks, scenes, prompts, and decoding settings across erasers. An eraser "matches real absence" if its per-model hidden-hallucination rate falls within a paired-bootstrap confidence interval of the physical-removal rate. The noise-patch condition stands in for the CofactGVR-style intervention. CofactGVR reports 90.0 on POPE (CofactGVR), but no reported result was found on Lacuna for any model on physically removed objects. Our own measurements are therefore the first reference numbers there.
For the mitigation question, the competitors are LLaVA-1.5-7B with and without oDPO, and the CofactGVR-trained model if weights are available. Each is evaluated on identical items. Suppose oDPO's acc+ gain on LaMa-erased fresh scenes is significantly larger than its gain on physically removed versions of the same scenes. We would read that as the fix partly learning eraser signatures.
The key ablation isolates physical truth itself. We rerun every analysis replacing the physical "without" image with the best reference-free prediction of cleanliness: the eraser output that scores best on ReMOVE, CFD, and MSN. If these metrics select edits on which models behave as they do on real absence, the physical set is unnecessary for future benchmarks.
Why not just…?
Why not just use the cleanest eraser and trust it? This is the most serious alternative, and our design tests it directly. EntityErasure's low sundry count shows it adds fewer phantom objects. But MSN counts new objects, not residual shadows or texture seams of the removed object, and EntityErasure itself reports failures on heavy occlusion and shadows. If its edits match physical absence within the confidence interval, we will say so. The practical recommendation would then be to rebuild ROHE-style benchmarks with it, and the physical set would serve only as a one-time certificate.
Why not just paint a black box or noise over the object? A box tests whether the model notices a corruption, not whether it perceives absence. Zhang et al. show that synthetic masks are easy to recognize as artifacts. We keep the noise condition anyway, because the size of the gap between it and physical removal quantifies how much a CofactGVR-style signal differs from real absence.
Why not just filter contaminated COCO items, or use only new photos? Fresh photos remove the memorization effect but not the eraser effect, so they solve one confound out of three. Filtering by n-gram or loss overlap does not apply cleanly to images, and DyVal's results suggest static decontamination misses conceptual leakage. Our fresh-versus-COCO comparison at a fixed eraser directly measures how much filtering would buy.
Why not just run an open-vocabulary detector on the erased image and discard items where it still finds the object? This cheap filter is essentially an MSN-style check, and we include it as a condition. Its weakness is that detectors and the tested models share visual encoders and blind spots, so an item a detector passes may still carry traces the model uses. The physical set measures how many such items slip through.
Proposed Experiments
The decisive check is a within-scene comparison. For each model, eraser, and fresh scene, we compute the hidden-hallucination rate on the erased image minus the rate on the physically removed image. This eraser-trace share is reported per eraser, across open models (LLaVA-1.5-7B/13B, InternVL via its public release) and any accessible closed model.
The second check is a difference-in-differences between COCO and fresh images at a fixed eraser, stratified by the recall probe. It estimates the memorization share, and it tests whether items the model recalls from blurred input carry more hallucination.
The third check correlates ReMOVE, CFD, MSN, and a full-reference score against the physical "without" image with per-item model fooling. This identifies which cleanliness metric predicts benchmark validity. It also turns the long-standing "no ground truth" problem into a measurable calibration.
The fourth check repeats the oDPO and CofactGVR comparisons on physical versus erased items.
The fifth check runs the multi-step question set. We measure whether a presence error propagates to the downstream answer, how that rate differs between erased and physically removed conditions, and whether step-by-step prompting widens the gap.
The proposal can succeed in two ways. The first is a clear decomposition showing that existing removed-object scores materially over- or under-state hallucination. The second is a certificate that a specific eraser reproduces real absence, which would validate that eraser for future benchmarks.
References
- Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects?
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- EntityErasure: Erasing Entity Cleanly via Amodal Entity Segmentation and Completion
- A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects
- Object Remover Performance Evaluation Methods Using Classwise Object Removal Images
- ReMOVE: A Reference-free Metric for Object Erasure
- OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
- VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context
- CofactGVR: Counterfactual Intervention for Grounded Visual Reasoning
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
- Text-Guided Image Inpainting