Seen Then Hidden: Teaching Video Language Models When an Unseen Object Is Still There

Vision-language models are now trained, and patched after training, to say only what the pixels support. That rule breaks under occlusion. A person who watches a dog walk behind a couch knows the dog is still there, even though no pixel shows it. Current anti-hallucination methods would count that belief as a hallucination. The static-image occlusion benchmark O-Bench finds that multimodal models already lean too far this way: they answer "no" about hidden objects unless the visible evidence is overwhelming (Beyond the Visible). Video offers a principled way out. It records whether the model has actually seen the hidden object before. An assertion about something now hidden is justified if it was visible earlier. It is a hallucination if it rests only on what usually co-occurs with the scene.

We propose to make that distinction trainable and measurable. We build synthetic occlusion videos in counterfactual pairs. In both videos of a pair, every frame after the occluder arrives is pixel-identical. The two differ only in whether the object was visible before it was covered: in one it was, in the other it never was. In a third variant it was visible, but the occluder then plainly removes it, for example by carrying it away. Frame-level evidence cannot tell these videos apart, so a model can only answer correctly by carrying information forward in time. We fine-tune a token-efficient video language model on these pairs. We then test a specific failure of its two-speed visual encoder: the brief window when an object is visible is easily lost when the slow, high-resolution pathway samples frames sparsely.

Core Claim

Hallucination under occlusion is not one quantity. It splits into two errors that current methods conflate. The first is denying an object the model saw become occluded, which is an amodal failure. The second is asserting an object the model never saw, which is a true hallucination. Existing mitigations push both errors in the same direction. Generic synthetic-occlusion fine-tuning without counterfactual pairing will also reduce the first error mostly by raising the "yes" rate. That rise shows up as a larger second error.

We claim that paired videos, in which only the model's memory of the past can decide the answer, reduce both errors at once. Where the video encoder is the limiting factor, the remaining failures should follow a simple rule: accuracy on the "seen then hidden" condition should collapse when the pre-occlusion visibility window is shorter than the slow pathway's frame stride. If both of these predictions hold, a video model's hallucination behavior can be tuned toward object permanence without making it gullible.

Positioning Against Prior Work

The motivating evidence comes from three sources. O-Bench builds layered, depth-aware synthetic occlusion images. It shows that scaling the language model from 7B to 72B gives diminishing returns and that the vision encoder is the bottleneck. The best model reaches an O-Score of 62.2% against 89.6% for humans (Beyond the Visible). O-Bench is image-only and diagnostic. We move to video and add a training intervention. PhysMLLMs reports that video language models lose targets after occlusion and responds by distilling DINOv2 features. Its gain on ReVOS occlusion and disappearance cases is +1.92 J&F, and it still fails on persistent occlusion (PhysMLLMs). That work regularizes features globally. It never asks the model to reason about whether a hidden object exists.

Two hallucination papers constrain the design. POPE shows that presence questions are dominated by co-occurrence priors and a strong "yes" bias (Evaluating Object Hallucination). Any occlusion-training scheme that simply teaches "hidden things exist" risks re-creating that bias. We therefore report POPE-style adversarial negatives inside occlusion scenes. SHARP steers middle-layer activations away from states with little visual support (SHARP). By construction it should suppress correct amodal answers, and we test that prediction directly.

Two sources supply the method. The compositing machinery follows the Object-video-Overlay (OvO) pipeline from TACO, which pastes object tracks over video backgrounds with controlled occlusion ratio, edge feathering and persistent occlusion (TACO). TACO uses that data to generate pixels. We use it to generate questions whose answers depend on visibility history. The insight that time supervises amodal inference comes from SaVos, which fills in occluded parts using pixels visible in other frames (SaVos). SaVos works on masks with shallow networks. We apply the same principle to language answers.

The model under study is SlowFast-LLaVA-1.5, which encodes 128 frames in about 9K tokens. It uses a slow pathway (few frames, high spatial resolution) and a fast pathway (many frames, heavily pooled) (SlowFast-LLaVA-1.5). Its authors note that uniform sampling can miss brief events. We turn that remark into a measurable prediction about occlusion.

Research Program

Paired videos. We start from SA-V and MVImgNet object tracks as OvO does, and from SAIL-VOS clips, where the game engine provides exact amodal masks and visibility (SAIL-VOS). From these we render triples that share an identical post-occlusion segment.

In the seen-then-hidden variant, the target is visible for a controlled window of $w$ frames and then occluded, with the occluded fraction swept up to full cover. In the never-seen variant, the target is absent throughout, but the occluder and the scene context are chosen to make the target a likely co-occurrence guess. We take the idea of GPT-4o-suggested plausible pairings from O-Bench and invert it to choose tempting absent objects. In the seen-then-removed variant, the target is visible and then carried out of frame by the occluder, or visibly passes through it.

Questions cover three things: presence ("Is there a cup behind the box?"), attributes of the hidden object (color and category), and amodal extent (roughly what fraction is hidden). The last extends O-Bench's occlusion-rate task, where models average 27.3%. The answer space includes "cannot tell" for questions about never-seen regions.

O-Bench's human filter discarded over 96% of generated images. We sidestep most of that cost because our label comes from the rendering script, not from realism judgments. We human-filter only a held-out evaluation split.

Training. We fine-tune SlowFast-LLaVA-1.5 (1B and 7B) with low-rank adapters on the projector and language model. We mix the paired data with the model's original video-image instruction data so that general benchmarks can be monitored.

Encoder analysis. We sweep $w$ against the slow pathway's stride $s$. We also sweep target size, because the fast pathway's pooling may erase small objects. If the "stride rule" appears, we test one minimal fix: add slow-pathway frames at moments of large visual change. This is a detector-free heuristic, and we run it at a matched token budget.

We summarize behavior with two rates on the paired evaluation set. $R_{\text{deny}}$ is the fraction of seen-then-hidden presence questions answered "no" or "cannot tell". $R_{\text{halluc}}$ is the fraction of never-seen and seen-then-removed questions answered "yes". A method improves only if it lowers both; trading one for the other does not count.

Baselines

The do-nothing reference is SlowFast-LLaVA-1.5 used zero-shot at its released settings. Its reported general-video numbers are 62.5% on LongVideoBench and 71.5% on MLVU for the 7B model (SlowFast-LLaVA-1.5). We re-measure these to show that our fine-tuning does not regress them by more than one point. No reported result was found on Lacuna for any video language model on a video occlusion benchmark, so the zero-shot $R_{\text{deny}}$ and $R_{\text{halluc}}$ are measured here. Our prediction from O-Bench is a high $R_{\text{deny}}$.

The strongest competitor on the hallucination side is SHARP, applied at the paper's middle layers. We derive its steering vectors from its own stimulus data and, separately, from our never-seen videos. Its reported gain is POPE adversarial accuracy rising from 79.5% to 82.8% on LLaVA-Next (SHARP). We expect it to lower $R_{\text{halluc}}$ while raising $R_{\text{deny}}$. Beating it means a lower $R_{\text{deny}}$ at an equal or lower $R_{\text{halluc}}$.

RITUAL is applied per frame at matched total compute, since it needs two forward passes (RITUAL). It serves as the training-free augmentation reference; its reported POPE gain on LLaVA-1.5 is from 84.13% to 88.87%.

The strongest competitor on the occlusion side is PhysMLLMs' DINOv2 distillation. We reproduce it on the Sa2VA-InternVL3-2B base that paper uses and also transplant it onto our base. We compare on the ReVOS occlusion and disappearance subset, where its reported gain is +1.92 J&F, and on our paired set.

The ablation that isolates our key ingredient is fine-tuning on the same number of synthetic occlusion videos, with the same questions, adapters, steps and learning-rate sweep, but without pairing. In this condition, seen and never-seen examples come from different videos, so frame content can partly predict the answer. A second ablation trains on static O-Bench-style occlusion images to test whether video matters at all. All fine-tuned variants receive the same hyperparameter search budget of 8 configurations, selected on a validation split disjoint from the evaluation clips.

Why not just…?

Why not just fine-tune on lots of synthetic occlusion video without the pairing? This is the real threat. It is included as the central ablation, not dismissed. Our prediction, based on POPE's finding that presence answers slide toward co-occurrence and "yes", is that unpaired training lowers $R_{\text{deny}}$ largely by raising $R_{\text{halluc}}$ on the tempting never-seen videos. If unpaired training matches paired training on both rates, the pairing adds nothing. We would report that and reframe the contribution around the benchmark and the stride analysis.

Why not just scale the language model? O-Bench found diminishing returns from 7B to 72B on occlusion perception (Beyond the Visible). In our paired videos, the post-occlusion frames are identical, so no amount of world knowledge can separate seen from never-seen without memory of earlier frames. We still run the 1B and 7B models to show whether the gap between the two rates shrinks with scale.

Why not just prompt the model to reason step by step about what is hidden? Longer generated reasoning shifts attention from visual tokens to the model's own text (Why LVLMs Are More Prone to Hallucinations in Longer Responses). Models also often produce correct answers with fabricated justifications (Yan et al.). We include a chain-of-thought prompt baseline on the zero-shot model. If a prompt closes most of the gap, the encoder is not the bottleneck and our stride analysis becomes the main result.

Why not just sample more frames densely? Dense high-resolution sampling blows past the token budget: competitors spend over 14K tokens on 64 frames where SlowFast-LLaVA-1.5 spends about 9K on 128. Still, the stride rule predicts that denser slow sampling should fix short-window failures. So we report a dense-sampling upper bound on short clips. Any fix we propose must approach that bound at the matched 9K budget.

Decisive Checks

The first decisive check is the paired evaluation. Paired training must lower $R_{\text{deny}}$ relative to the zero-shot model and SHARP without raising $R_{\text{halluc}}$, and it must beat the unpaired ablation on the never-seen condition specifically. The second is the stride sweep. If failures on seen-then-hidden clips concentrate where $w < s$ and vanish under change-triggered slow frames at equal budget, we have located a concrete encoder-level cause for occlusion hallucination in two-speed video models. The third is transfer out of synthetic data. We look for gains on the ReVOS occlusion subset against PhysMLLMs, on O-Bench's static tasks against the published 62.2% top O-Score, on the agent-interaction questions of STSBench (STSBench), and for no loss on POPE adversarial. A method that passes the synthetic checks but fails all three transfer tests would indicate that it has learned the compositing artifacts rather than object permanence.