When people rate a streaming video, a brief glitch drags their opinion down at once, and their opinion recovers only slowly after the picture returns to normal. Video-quality researchers call this perceptual hysteresis: a system whose current state depends on its recent history, with a fast fall and a slow recovery. This proposal asks whether the same thing happens inside audio-visual language models, meaning large language models that take video frames and sound as input and answer questions in text. Concretely, suppose a short stretch of a long video has a sensory conflict, such as a soundtrack that belongs to a different scene. Does the model keep hallucinating about later, perfectly clean stretches, attributing the wrong sound to the wrong object long after the conflict has ended? If it does, then part of multimodal hallucination is a memory problem and not only a perception problem. Current fixes adjust each output token in isolation, and they cannot reach an error that lives in the model's context.
The plan has three parts. We build a controlled long-video probe set by splicing short audio-visual clips into streams and inserting conflict windows at known times. We measure the model's error on later clean clips as a function of time since the conflict, and we fit the same low-order state-space models that quality researchers use for human memory of impairments. We then test a training-free fix: detect the conflicting window with a local audio-visual agreement score, and at answer time contrast the model's output with and without that stretch of history. The probes ask where and which, meaning which visible object is making the sound, and not only whether. Our hypothesis is that contamination shows up first at the level of local grounding, while clip-level yes/no existence questions still look fine.
Core Claim
Hallucination in long audio-visual inputs has a history-dependent part. A transient cross-modal conflict raises error on subsequent clean segments. That excess error decays with a measurable time constant, the decay is asymmetric (onset is fast, recovery is slow), and it is larger for region-level grounding questions than for existence questions. Because this part of the error is caused by specific earlier context, it should be removable by counterfactually masking that context. That fix should beat both per-token contrastive decoding, which does not target history, and the blunt choice of throwing away all history, which removes useful long-range context along with the contamination.
The claim can be falsified. If the excess error on clean clips vanishes within one clip of the conflict ending, the models have no meaningful hysteresis, and the mitigation half of this proposal should be dropped. A clean negative result of that kind would still tell the community that long-context audio-visual hallucination is not a carry-over effect.
Where This Sits
The measurement tools come from the quality-assessment literature. Modeling Continuous Video QoE Evolution: A State Space Approach models continuous human quality ratings with an input nonlinearity followed by a third-order linear state-space system, and reports an LCC of 0.83 and an SROCC of 0.79 on LFOVIA. Quality Assessment of In-the-Wild Videos shows that pooling over time with asymmetric weights, which penalize bad frames more heavily, beats learned symmetric temporal pooling. Neither paper touches semantic errors or language models. We reuse their state-space form as a descriptive instrument for a model's error trace, not as part of the model itself. Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis motivates the idea that hysteresis emerges in networks that carry state. It shows recurrent vision networks whose decisions depend on the direction of a morph sequence. A decoder that attends over its whole context carries state in a similar way.
The motivating evidence for the hallucination side comes from three papers. AVHBench shows that audio-visual models hallucinate across modalities: they see things they only hear, and they hear things they only see. They often perform worse with both modalities than with one, and X-InstructBLIP scores 18.1% on audio-driven video hallucination. However, AVHBench uses roughly 10-second clips and clip-level presence questions, so it cannot observe carry-over. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that hallucinations in long outputs are driven by the model attending to its own earlier context rather than accumulating at random. That supports a history-dependent mechanism, but the evidence is for static images and generated text, not for input history. SmartSight links video hallucination to attention collapsing onto a single frame and prunes the sampled candidates that collapse. It treats the failure as symmetric over time and uses vision only.
The methods we compare against or adapt are these. AVCD contrasts clean logits with logits computed under audio-masked, video-masked and dual-masked conditions. It masks by modality and attention dominance, not by time window. We borrow its contrastive arithmetic but mask along time instead. Dr.V verifies video answers with external detectors and temporal grounders, but it ignores audio and is expensive. Its bounding-box and timestamp annotations help us build localization probes. Look, Listen, and Answer constrains our design. It shows that answers on MUSIC-AVQA can be guessed from the question alone more than 54% of the time. Every probe we build therefore needs a question-only control. Evaluating Object Hallucination in Large Vision-Language Models supplies the yes/no polling format with adversarial negatives. It also warns about the strong "yes" bias, which we control by balancing answers.
Research Program
Probe streams. We build 2–5 minute streams by concatenating short real clips drawn from the sources AVHBench uses (VALOR, AudioCaps). A random subset of streams contains one conflict window. The window is either an audio swap taken from AVHBench's audio-mismatched clips, or a visual degradation (heavy blur or occlusion) with the audio left intact. The window's duration and position are varied. Matched control streams are identical except that the window is clean.
After each stream, the model answers probes about a single target clip $k$ clips after the conflict. The probes come in three kinds. Existence probes follow the POPE format ("is there a dog in segment 7?"). Local grounding probes ask which of several region proposals contains the sound source in that segment. The proposals come from an open-vocabulary detector in the style of GLIP, with boxes checked by humans on a subset. Cross-segment probes need genuine long-range context, for example "is the instrument heard in segment 7 the same one shown in segment 2?" Every probe type also gets a question-only control and a single-clip control, which shows the model only the target clip.
Measuring hysteresis. For each model we compute the excess error $e(k)$ on clean target clips, defined as error in conflict streams minus error in matched controls, as a function of the distance $k$ after the conflict window. We also compute the mirror quantity for clips before the window, which controls for the model reasoning backward through the stream. We fit the third-order state-space form from Eswara et al. to the per-clip error trace, with conflict presence as input. From the fit we report a recovery time constant and an onset/recovery asymmetry ratio. We compare these quantities between audio-conflict and visual-conflict windows, and between existence and grounding probes.
Mitigation by counterfactual history masking. For each window $w$ of history we compute a local audio-visual agreement score $s_w$. This is the best cosine similarity between the window's audio embedding and any region embedding in its frames, computed with a shared tri-modal embedding space of the CLIP4VLA kind. The score is local on purpose: a whole-frame score would miss off-screen or mislocated sources. Windows with $s_w$ below a threshold $\tau$ are flagged. When answering about a later segment, the model produces logits $\ell$ on the full context and logits $\tilde{\ell}$ with the flagged windows' tokens masked. Decoding then uses
$$ \hat{\ell} = (1+\alpha)\,\tilde{\ell} - \alpha\,\ell , $$
where $\alpha \ge 0$ sets how strongly we move away from the history-contaminated prediction. Here the masked pass is the trusted one and the full pass is the biased one, which reverses the roles in AVCD's modality masking. We tune $\tau$ and $\alpha$ once on a held-out split of streams and then freeze them. The method needs no training and costs one extra forward pass per answer.
Baselines
The naive reference point is the unmodified model on the full stream. Its difference from the matched clean control defines the effect we are trying to remove. The published reference numbers are 18.1% A→V accuracy for X-InstructBLIP and "slightly better than random" for most baselines on AVHBench. On our long streams there is no reported result, and no reported result was found on Lacuna for any model on long spliced audio-visual streams.
The strongest existing training-free competitor is AVCD. It reports overall AVHBench accuracy gains of 2% for VideoLLaMA2 and 7% for video-SALMONN. It is the strongest competitor because it is the only published decoding-time method built for audio-visual hallucination, and it already uses masked counterfactual passes. The only difference from our method is the masking axis: modality in AVCD, time window in ours. We run it with the authors' settings on the same two models and the same streams. Our method must beat it on post-conflict clean clips by a margin larger than the seed-to-seed variation. It must also not lose to AVCD on AVHBench itself, where there is no long history to exploit.
SmartSight reports a 10.59% F1 gain on VRIPT-HAL for Qwen2.5-VL-7B. We run it on the visual-degradation arm, where audio does not matter, with the same number of sampled candidates as our compute budget allows. It tests whether pruning collapsed candidates already fixes carry-over.
The ablation that isolates our key ingredient, which is choosing which history to mask, has three parts. First, masking the same number of randomly chosen windows. Second, masking with an oracle that knows the true conflict window, which gives the upper bound. Third, masking all history except the target clip, which is the chunking baseline discussed below.
The trained competitor is LoRA fine-tuning on perturbed streams. AVHBench reports that LoRA on its training split moved Video-LLaMA from about 50% to 83.9% on A→V. We give it the same perturbed training streams used to tune $\tau$ and $\alpha$, plus a compute budget matched to a full sweep of our method. For the tool-based upper reference we run Dr.V Dr.V-Agent on the visual arm and report its latency alongside accuracy. It reports +18.60% over Self-PEP for VideoChat2 on Dr.V-Bench.
Why not just…?
Why not just answer each segment in isolation? If carry-over is the problem, cutting the history removes it by construction. This is a serious competitor on the probes about single clean clips, and we expect it to match our method there. Its cost is the cross-segment probes, which need long-range context, so the proposal stands or falls on the joint score. Our method has to match chunking on contaminated single-clip probes while staying close to full-context performance on cross-segment probes. If chunking wins on both, the right conclusion is that current models gain nothing from history, and we will say so.
Why not just use AVCD or another per-token contrastive decoder? AVCD's masked passes remove attention-dominant tokens within a modality across the whole input. They do not single out an earlier time window whose content is still shaping the answer. Its entropy gate also skips tokens where the model is confidently wrong, which is exactly what we expect from carry-over. This is an empirical claim. The comparison on post-conflict clips settles it, and so does the check of whether AVCD's gain grows or shrinks with the distance $k$ after the conflict.
Why not just fine-tune on perturbed streams? AVHBench's jump to 83.9% after LoRA shows that fine-tuning is powerful, and it may well beat a training-free method on the perturbation types it was trained on. We therefore test it on held-out perturbation types. It is trained on audio swaps and tested on visual occlusion, and the reverse. We also check whether it hurts the cross-segment probes by teaching the model to ignore history in general. If fine-tuning generalizes across perturbation types and keeps long-range context, it is the better practical answer. The hysteresis measurements would remain the contribution, and the mitigation would be secondary.
Proposed Experiments and Decisive Checks
The main models are VideoLLaMA2 and video-SALMONN, the audio-visual models AVCD was evaluated on. Qwen2.5-VL-7B serves the visual-only arm. We report accuracy for existence and localization probes, balanced so that "yes" and "no" are equally often correct, with the "yes" rate reported separately. From the state-space fits we report the recovery time constant and the asymmetry ratio. We also report extra forward passes per answer.
The first decisive check is whether $e(k)$ stays above zero for $k>1$ at all. The significance test uses paired conflict/control streams, and the question-only and single-clip controls rule out priors and target-clip difficulty.
The second check is whether localization probes show a larger or longer-lasting excess error than existence probes. This is the specific prediction that ties local grounding to carry-over.
The third check concerns direction. We compare whether an audio conflict contaminates later visual grounding more than a visual degradation contaminates later sound attribution. Given AVHBench's finding that sight and sound interfere with each other, we expect the two directions to differ.
The fourth check tests the mitigation against oracle, random and all-history masking, judged on the joint single-clip and cross-segment score. We also measure how often the local agreement detector correctly flags the conflict window, since a weak detector would limit the method more than the contrastive step does.
Finally, we run external-validity checks: natural, non-spliced audio-visual discontinuities in longer videos of the kind in Dr.V-Bench's ActivityNet and YouCook2 sources, and AVHBench and MUSIC-AVQA-R themselves, to confirm the method does no harm when there is no long history.
References
- Modeling Continuous Video QoE Evolution: A State Space Approach
- Quality Assessment of In-the-Wild Videos
- Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
- Dr.V : A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-Grained Spatial-Temporal Grounding
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering
- Evaluating Object Hallucination in Large Vision-Language Models
- Grounded Language-Image Pre-training