Neighboring Windows as Free Counterfactuals for Streaming Video Hallucination

Streaming video assistants watch a live feed through a short sliding window, for example the last four frames at one frame per second, and answer questions about what is happening now. When the answer is not visible in that window, they still answer, and they answer with whatever a typical scene would contain. Snow becomes "skiing," and a kitchen becomes "cooking." This note proposes a cheap test for that failure. A sliding-window system already holds the windows that came just before the current one. These windows show the same scene at slightly different moments. If the model's support for an answer does not change when the window moves to frames that lack the claimed event, the answer comes from the scene or from language habits, not from the video. We use that change, rather than agreement across windows, to rerank answers and to decide when to say "I don't know."

The timing matters because the field is moving toward simple memory-free windows. StreamOPD shows a 4-frame causal window can reach 83.9% on StreamingBench with a 4B model. The same paper reports that its training makes the model overconfident: the Hallucination Detection (HLD) score on OVO-Bench drops, meaning the model gets worse at abstaining. The best training-free fixes, contrastive decoding methods, contrast the real video against a deliberately corrupted copy. They were built for offline clips and need an extra full forward pass. Overlapping windows already provide the corrupted copy at no extra cost: the earlier window is a natural, scene-matched version of the input with the newest frames removed.

Core Claim

In a streaming video model with overlapping windows (window length $W$, stride $S<W$), the cached visual states of the preceding windows can serve as negative examples for contrastive scoring. These negatives keep the scene fixed and change only which moments are visible. Scoring candidate answers by how sensitive they are to the current frames does three things. It suppresses event and action-scene hallucination about as well as synthetic frame-dropping contrast. It restores abstention on questions whose evidence is absent. And it adds only text-token computation, because the visual features of past windows are already computed and stored.

A second claim is a falsifiable prediction that sets the proposal apart from classical temporal averaging. Prior-driven answers are the most stable answers across windows, because the prior does not move. So majority voting over overlapping windows should make hallucination worse on prior-conflict items, while sensitivity-based scoring makes it better.

Mechanism

Let $q$ be the query arriving at time $t$, $a$ a candidate answer, $V_t$ the current window, and $\mathcal{C}_t = \{V_{t-1}, \dots, V_{t-K}, \varnothing\}$ a contrast set. The contrast set holds the $K$ most recent earlier windows, already stored in the key-value cache (the per-layer attention states a transformer keeps so it does not recompute past inputs), plus an empty visual input. Define the contrast probability $\bar p(a) = \frac{1}{|\mathcal{C}_t|}\sum_{c\in\mathcal{C}_t} p(a \mid c, q)$, the model's average belief in $a$ across the contrast inputs. The reranking score is

$$s(a) = \log p(a \mid V_t, q) - \alpha \log \bar p(a),$$

where $\alpha \ge 0$ sets the contrast strength. This follows the contrastive decoding form of Leng et al.. Candidates are restricted to those with $p(a\mid V_t,q) \ge \beta \max_{a'} p(a'\mid V_t,q)$, the adaptive plausibility constraint of Im et al..

When the stream is static, every window gives the same probabilities and $s(a)$ reduces to $(1-\alpha)\log p(a\mid V_t,q)$. The ranking is unchanged for $\alpha<1$, so static scenes are not penalized. The model abstains when no candidate gains support from the current frames, that is, when $\max_a [\log p(a\mid V_t,q) - \log \bar p(a)] < \tau$ for a threshold $\tau$ tuned on held-out data.

Each term $p(a\mid c,q)$ needs only a forward pass over the query and answer tokens, attending to the cached visual states of window $c$. The visual prefill, which dominates cost at hundreds of tokens per frame, is never repeated. For open-ended generation the same contrast applies token by token.

The overlap ratio $(W-S)/W$ is the main knob from the windowing literature. Higher overlap gives more contrast windows that differ in fewer frames, so the counterfactuals are finer but weaker. We study it directly.

Relation to Prior Work

Two papers motivate the gap. EventHallusion shows that Video-LLMs fabricate events from priors. HAVEN attributes about 75% of video hallucinations to "prior conflict." StreamOPD shows the abstention collapse in exactly the 4-frame streaming setting we target.

The method comes from Temporal Contrastive Decoding (TCD), which contrasts a full clip against a frame-dropped copy. We replace the synthetic frame-dropped copy with real neighboring windows, move the setting from offline to streaming, and add an abstention rule that TCD does not have.

MASH-VLM attacks action-scene hallucination by redesigning attention, which requires fine-tuning on 430k examples. Our approach needs no training. MASH-VLM's UNSCENE scene-only videos are a natural stress test for us: the scene is constant across windows and the action never happens, so the contrast should cancel the prior.

Several papers constrain the design. RITUAL uses spatial augmentations; VidHal shows that models make identical temporal errors from a single frame and from the full video, which suggests spatial augmentation leaves temporal priors untouched. TriCD gains over 10% on OmniVCHall but needs three passes over perturbed video, which is the latency cost we aim to avoid. Temporal-aware activation engineering steers attention heads with vectors computed offline and selects them by classifying a whole clip as static or dynamic. In a stream that label changes from window to window.

Finally, the classical windowing work, video colorization by Kouzouglidis et al. and reflection removal by Nandoriya et al., averages continuous outputs across overlapping windows to suppress noise. We argue that for discrete, prior-laden answers the useful signal across windows is variation, not the average.

Research Program

We build the scorer on public backbones that some baselines also report on. LLaVA-NeXT-Video gives comparability with TCD on EventHallusion, and Qwen2.5-VL is used in the activation-engineering paper. Each runs in a StreamOPD-style memory-free causal window with $W=4$ frames at 1 fps and stride $S=1$. We also apply the scorer to StreamOPD-trained checkpoints if they are released, since their HLD drop is the clearest target.

The streaming harness comes from OpenGVLab/RIVER. Offline hallucination benchmarks (EventHallusion, VidHal, UNSCENE) are converted to streams by feeding each clip at 1 fps through the same window, with the query posed at the clip's end. Short clips yield few contrast windows, and we report the effect of $K$ separately.

The analysis has three parts. The first measures cross-window sensitivity directly on HAVEN items labeled as prior conflict versus capability deficiency. We predict sensitivity is lower on prior-conflict errors. The second sweeps the overlap ratio and $K$. The third characterizes failures. The main risk is an action that spans every cached window: its correct answer gains nothing over the contrast set and is wrongly abstained on. We report false abstention on answerable items as a primary metric, not a footnote.

Baselines

The naive reference point is the same backbone in the same 4-frame window with greedy decoding and no intervention. Every gain is measured against it. For EventHallusion the reference numbers are the base accuracies reported by EventHallusion: 64.79% for LLaVA-NeXT-Video and 63.33% for VILA.

The strongest existing training-free competitor on event hallucination is TCD, applied per window with its own frame-dropped extra pass. It is the strongest competitor because it targets the same prior-driven event errors with the same contrastive form. The difference is that it pays a full extra visual prefill instead of reusing the cache. Its reported results are 67.48% on EventHallusion for LLaVA-NeXT-Video (up from 64.79%) and 66.50% for VILA, both from EventHallusion. We run TCD on identical streamed inputs and give $\alpha$ and $\beta$ the same tuning budget on the same held-out split. We report accuracy against both wall-clock latency and visual-token FLOPs. Beating it means matching its accuracy within noise at under 1.2 times the naive latency, or exceeding it at equal latency.

On compositional hallucination the strongest competitor is TriCD. It reports a 9.61% improvement on OmniVCHall for Qwen3-VL-Instruct-8B and 12.03% for VideoLLaMA3-7B, per its paper. We compare on OmniVCHall at matched backbone. We expect to trail TriCD in accuracy, and the question is how much accuracy each extra pass buys.

For abstention, the strongest reference is the HLD score of the StreamOPD student. For that we found no reported numerical HLD result on Lacuna, only the statement that HLD decreased. We therefore re-measure it under our harness rather than quote it. For streaming accuracy the reference is StreamOPD's 83.9% on StreamingBench, and our requirement is that the scorer not reduce StreamingBench accuracy by more than one point when applied on top.

The key ablation swaps the contrast source while keeping everything else fixed. One arm uses only the empty visual input, which isolates language prior. One uses only the cached earlier windows, which isolates scene prior. A third uses synthetically frame-dropped versions of the current window. If the cached-window arm does no better than the empty-input arm on UNSCENE and on EventHallusion's Mix Common-Rare split, the windowing ingredient adds nothing.

Why not just…?

Why not just take a majority vote across the overlapping windows, the way temporal averaging removes flicker? Voting rewards stability, and prior-driven answers are the most stable answers in a stream. The scene that triggers "skiing" is present in every window. We treat this as a prediction to test: voting will raise accuracy on capability-deficiency items, where it removes noise, and lower it on HAVEN prior-conflict items and UNSCENE scene-only videos. If voting instead helps on prior-conflict items, our central argument is wrong, and the experiment would show it.

Why not just run TCD in every window? This is the real threat. TCD already works, with about a 2.7-point gain on EventHallusion for LLaVA-NeXT-Video. Synthetic frame-dropping may also produce cleaner counterfactuals than natural neighboring windows. Our case rests on cost and on abstention. Each TCD query repeats the visual prefill, and TCD has no rule for saying "I don't know." The matched-latency comparison settles the cost question. If TCD wins at equal latency, the contribution reduces to the abstention rule, which can be bolted onto TCD, and we would report it that way.

Why not just abstain when the model's own confidence or entropy is low? The models are confidently wrong on exactly these items. POPE reports "Yes" rates near 99% in some settings, and StreamOPD's distillation increases confidence. An entropy threshold will therefore keep prior-driven answers. We include entropy-based abstention, tuned on the same held-out split, as a baseline on OVO-Bench HLD. Our rule must beat it at matched false-abstention rates on answerable questions.

Why not just widen the window or add a memory module? RIVER shows that memory slows forgetting, with a 12% shallower decay slope. More context does not remove the scene prior, however, and long-response analyses show that longer contexts shift attention away from visual tokens. The scorer is complementary to memory. We test it on RIVER's memory-module configuration, as well as the memory-free one, to check that the gains add.

Proposed Experiments and Decisive Checks

The hallucination suite measures accuracy against latency on EventHallusion (all three splits), VidHal (multiple-choice accuracy and the caption-ordering relative NDCG), UNSCENE binary tasks, and OmniVCHall. The streaming suite measures StreamingBench accuracy, OVO-Bench accuracy and HLD, and RIVER's Live-Perception, Retrospective and Proactive tasks. All comparisons use the same backbones, windows and tuning budget.

Three checks decide the proposal. First, on UNSCENE scene-only videos and EventHallusion's Entire Rare split, the scorer must reduce affirmative errors relative to the naive window, and the cached-window contrast must beat the empty-input contrast. This shows that scene-matched negatives carry signal that language-only contrast lacks. Second, sensitivity scoring and majority voting must diverge in opposite directions on HAVEN prior-conflict items. Third, on OVO-Bench, HLD must recover relative to the uncorrected model while StreamingBench accuracy stays within one point, at under 1.2 times the naive per-query latency.

A secondary, exploratory arm borrows deferred decision-making from Lu et al.. For "has this state change happened?" captions, a claim is withheld until $k$ later windows show increasing sensitivity. We evaluate this on FrameCapEval against ProgressCaptioner. It is reported only if the main checks pass.