Do VLMs See Removed Objects Because of Pixels or Because of Memory?

When a vision-language model (VLM, a language model that also reads images) is shown a photo in which an object has been digitally erased and still reports the object as present, two very different things could be going wrong. The model may be reacting to leftovers of the erasure, such as faint outlines, smeared textures, or spurious new objects that the image-filling tool invented. Or it may be ignoring the pixels and relying on what it already "knows", either from scene co-occurrence ("kitchens have fridges") or from earlier frames and earlier answers in the conversation that mentioned the object. Current benchmarks cannot tell these apart. They test one edited image at a time and treat erasure as a clean switch from "present" to "absent", even though the erasure tools are known to leave artifacts.

We propose a controlled benchmark of sequential object removal that separates these causes. It varies three factors independently: how much residue the erasure leaves, how many removal steps the model has to track, and how much irrelevant clutter and how many irrelevant edits surround the relevant changes. The step that makes this possible is a simulator arm. There, the true background behind every removed object can be rendered, so we can produce perfectly clean removals and inpainted removals of the same object in the same scene. This also settles a problem that has blocked evaluation of object removal: there is normally no ground-truth background to compare against. With one, we can test which reference-free removal-quality score actually predicts downstream VLM errors.

Core Claim

The claim has two parts, and either could be false. First, part of the "still sees removed objects" failure reported on single images is caused by erasure residue rather than by language priors. That share should shrink sharply when clean simulator renders or cleaner erasers replace a basic inpainter such as LaMa. Second, in multi-step episodes the remaining failure is not explained by summing per-frame errors. Accumulated history acts as contextual noise: earlier frames and the model's own prior statements pull attention away from the current image. This history effect should interact more than additively with residue, because a faint ghost becomes decisive when the history already asserts the object exists.

If neither the residue effect nor the excess history effect appears, then single-frame benchmarks like ROHE are sufficient and the sequential framing adds nothing. That is a clean negative result worth reporting.

Grounding in Prior Work

Two benchmarks motivate the residue question. ROHE builds 5,504 COCO pairs by erasing objects with LaMa and scores models with acc+, which requires a correct answer on both the original and the edited image. MERLIM uses inpainting at a much larger scale and reports an approximately 8% precision drop on edited images. Both attribute the failure to the model and never measure the quality of the erasure itself.

Two other papers show that the erasure quality matters. EntityErasure documents that diffusion inpainters create spurious objects (sundries) in the erased region and introduces counts of them: Mean Sundries Num (MSN) and Mean Area Ratio of Sundries (MARS). Visual Jenga performs sequential removals and notes qualitatively that ghosting propagates errors down the sequence, but does not measure the effect. We reuse their tools and turn their observation into a controlled variable.

Three lines of work supply the measurement tools for residue, and all of them lack validation against a true background. Full-reference metrics are invalid here because the occluded background was never photographed (Mori et al.; Oh et al.). The reference-free alternatives are ReMOVE, which compares deep features of the filled patch against its surroundings; CFD, which scores coherence with the context and penalizes new object structures; and MSN/MARS. None has been checked against ground truth or against downstream VLM behavior.

The history question comes from state-tracking and noise work in other settings. ENACT finds that VLMs fall to near-random (about 47%) on 10-step world-modeling tasks while humans hold near 95%, and that they hallucinate transitions that never happened. ENACT uses clean simulator frames and does not vary clutter. CogniLoad shows in text-only puzzles that distractor density has a U-shaped effect and that task length and difficulty combine super-additively. We import CogniLoad's factorial design into vision. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows attention moving from image tokens toward previously generated text, which is the mechanism we expect history to trigger.

One finding constrains the design. EditInspector shows that VLMs judging edits hallucinate about twice as often on "Remove" as on "Add", and that VLM judges are lenient "yes-men". We therefore never use a VLM to score correctness. All labels come from simulator state or from segmentation masks.

Research Program

Each episode starts from a scene and removes $K$ objects one at a time, with $K$ ranging from 1 to 10. At every step the model sees the current frame and answers two kinds of questions. It answers presence probes for removed, never-present, and still-present objects, and it answers a cumulative question: "which objects have been removed so far?"

Four factors are crossed.

The first factor is the removal source. In the simulator arm, which uses the BEHAVIOR household scenes on which ENACT is built, the same removal is produced four ways: a true re-render without the object, EntityErasure, LaMa (to match ROHE), and a plain Stable Diffusion inpainter known to produce sundries. In the real-image arm, built on the COCO/ROHE source images, only the three inpainters are available.

The second factor is the history format. The model receives either the current frame alone, all prior frames, all prior frames plus its own previous answers, or the current frame plus a ground-truth text log of removals.

The third factor is distractor density. This is the number of irrelevant objects in the scene plus interleaved irrelevant edits, such as recoloring or moving an untouched object. It is the visual analogue of CogniLoad's needle-to-hay ratio.

The fourth factor is horizon $K$.

The central analysis compares each step against a single-frame reference. For each step $t$ we define the excess history error

$$\Delta_t = e_t^{\text{hist}} - e_t^{\text{frame}},$$

where $e_t^{\text{hist}}$ is the persistence-hallucination rate at step $t$ when history is shown, and $e_t^{\text{frame}}$ is the rate on the identical image shown alone. The "frame-only" condition is exactly a ROHE-style single-frame test. A positive $\Delta_t$ that grows with $t$ isolates a genuine history or state-tracking effect.

We then fit hallucination probability as a function of three things together: residue (measured against the simulator's true background), $t$, and distractor density. The interaction term tests whether residue and history combine super-additively.

The simulator arm also supports a metric study. For every inpainted removal we compute ReMOVE, CFD, MSN, MARS, and a full-reference distance to the true re-rendered background, which is valid here and nowhere else. We report two things for each metric: how well it agrees with the true-background distance, and how well it predicts whether a VLM will hallucinate the removed object. This tells practitioners which reference-free score to use when filtering counterfactual images on real photos, where no true background exists.

Baselines

The naive reference point is the frame-only condition. It is an ordinary single-image test on each step's frame, with no history, and it shows what per-frame perception alone produces. Every sequential condition is reported against it on identical images, so no condition gets easier inputs.

The strongest existing method on this question is object-based preference training (oDPO) from ROHE. It teaches the model to reject descriptions containing erased objects and is the only method trained specifically on removed-object negatives. ROHE reports that it lifts LLaVA-1.5-7B acc+ from 39.21% to 61.65%. We reproduce oDPO on LLaVA-1.5-7B with ROHE's data recipe. We first confirm that our reproduction recovers roughly that single-frame gain on ROHE, then evaluate it in our episodes with the same prompts and decoding settings as the base model. The question is whether oDPO's gain survives at long horizons and under history. If oDPO closes most of $\Delta_t$, the history effect is just a perceptual weakness that training already fixes. If $\Delta_t$ remains above zero for oDPO at $K\ge 5$, we have found something it does not address.

The strongest test-time baseline is RITUAL. It decodes jointly over the original and randomly transformed copies of the image and raises LLaVA-1.5 POPE accuracy from 84.13% to 88.87%. We run it on the current frame at every step. We match compute by giving every condition the same number of forward passes where possible, and we report latency.

The key ablation is the ground-truth text log. Giving the model a correct removal ledger, in the spirit of Statler's explicit world-state dictionary (40–55% success where Code-as-Policies scored 0%), removes the need to track state. If accuracy with the log matches frame-only accuracy, the remaining errors are perceptual. If the log does better than frame-only, the model can use correct memory but builds corrupted memory itself. A second ablation is clean simulator removal, which sets residue to zero. For horizon effects, the external reference point is ENACT's roughly 47% at 10 steps. We do not claim to beat it, since our task differs, but we check that our long-horizon accuracy falls in a comparable regime.

Why not just…?

Why not just use a simulator, where removals are perfectly clean? For the history question, that would mostly work, and it is why the simulator arm exists. But clean renders remove exactly the variable that real counterfactual benchmarks contain, so on its own the simulator cannot say whether ROHE-style numbers reflect the model or the eraser. Pairing clean and inpainted versions of the same removal is what turns the simulator from an alternative into a control.

Why not just use a cleaner eraser like EntityErasure and move on? EntityErasure reports MSN 0.122 versus 0.554 for PowerPaint and 0.679 for CLIPAway on EntitySeg. If swapping LaMa for it removes most hallucination, that is itself our first finding, and it would mean published removed-object hallucination rates are inflated. A low sundry count does not guarantee the absence of faint ghosts, though. The residue-versus-hallucination curve, computed against true backgrounds, shows whether the remaining residue still matters.

Why not just track state in text and skip vision? The text-log ablation is this alternative, and it is a real threat for practical systems. Two caveats apply. It assumes an oracle that knows which actions succeeded, which is exactly what Statler's authors note is missing when actions fail. And it cannot answer whether the model perceives the current image. If the text log plus the current frame solves the task, we will say so and recommend it, and the benchmark's scientific value then rests on the frame-only versus history comparison.

Why not just evaluate each frame independently with ROHE? If $\Delta_t$ is approximately zero everywhere, then per-frame evaluation is sufficient and the sequential benchmark is unnecessary. The design measures this directly instead of assuming the answer.

Proposed Experiments and Decisive Checks

Models are LLaVA-1.5-7B and 13B (matching ROHE), LLaVA-1.5-7B with our oDPO reproduction, and two proprietary models used in EditInspector, GPT-4o and Gemini 1.5 Pro. Metrics are persistence-hallucination rate, ghost-addition rate (reporting a sundry as a real object), per-step acc+, episode-level acc+ (every step correct), and $\Delta_t$.

The first decisive check is residue attribution. On matched simulator removals, compare the frame-only hallucination rate for clean renders against LaMa. If the gap is a substantial fraction of the LaMa error, single-image removed-object benchmarks partly measure the eraser.

The second check is the history effect. Test whether $\Delta_t$ increases with $t$ for every model, including oDPO, and whether it disappears with the ground-truth log.

The third check is the interaction. Test whether the residue-by-history interaction is positive, meaning ghosts are much more harmful when history mentions the object.

The fourth check is the noise curve. Test whether hallucination versus distractor density is U-shaped as in CogniLoad, or simply increasing, which would indicate a dilution mechanism different from text.

The fifth check is metric validity. Rank ReMOVE, CFD, MSN, and MARS by how well they predict VLM hallucination against the true-background distance. The best one is then used to stratify the real-image arm, showing whether the simulator conclusions transfer to COCO.

The result we expect most is a large residue share in the frame-only condition and a positive but smaller history effect that oDPO does not remove. Any other pattern is still informative because each factor is isolated.