Forgetting or Relapse? Separating Evidence Loss from Prior Override in Long-Video Models

When a video-language model answers a question about something it saw forty minutes ago and gets it wrong, the standard explanation is that it forgot. This proposal tests a different explanation. The model may still hold the evidence, but when that evidence contradicts what it learned in pretraining (a mug sitting under a table, a light object that stops a heavy one), a stubborn prior can override it. These two failures call for opposite fixes. Forgetting is fixed by putting the evidence back in view, for example by retrieving the relevant clip. Override is fixed by changing how the model weighs evidence against its prior, for example by steering its internal activations. Today's long-video benchmarks report one accuracy number, and that number cannot tell the two apart.

We propose a controlled "needle" design that can. Short clips that either agree or disagree with a model's prior are inserted at varying depths into hour-long videos. For each question, we also ask the model to recall the inserted evidence, in the same context. The central quantity is how often the model retains the evidence yet answers with its prior, and how that rate changes as the evidence recedes in time. If this rate rises with distance, then part of what the field calls temporal decay is a knowledge-conflict failure, and conflict-steering methods built for text should transfer to video in a specific, predictable way.

Core Claim

Temporal reasoning decay in long video has two separable components. The first is passive evidence loss: the evidence fades and the prior fills the vacuum. The second is active parametric relapse: the evidence is still recoverable but loses the internal competition with the prior. The second component should grow with temporal distance only for prior-violating content. We make three predictions:

  1. On items the model answers correctly when the clip is shown alone, accuracy on counter-prior needles falls faster with insertion depth than accuracy on matched prior-consistent needles.
  2. The extra errors land on the specific answer the prior favours, rather than spreading across distractors.
  3. A measurable fraction of those errors co-occur with successful recall of the evidence.

If any of the three fails, the simpler "it just forgot" account stands, and we will say so.

Where This Sits

Three bodies of work motivate the question.

Knowledge conflict. Mechanistic work shows that conflict between prompt and memory is detected before it is resolved. Analysing the Residual Stream of Language Models Under Knowledge Conflicts finds detection peaking around layer 14 of Llama3-8B with roughly 90% probe accuracy, and the resolution decision peaking later, at layers 16–17. Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models finds that image-text models are more stubborn than text models and grow more stubborn with scale, even while detecting the conflict 75–79% of the time. That is precisely the "notices but overrides" pattern we hypothesize for video. Both studies are static, so neither asks how the competition changes as evidence ages inside a long context.

Spatial priors. Spatial-reasoning work supplies the priors we exploit. What's "up" with vision-language models? documents an "on-table" world prior and near-chance spatial-preposition accuracy. ViewSpatial-Bench shows models defaulting to the camera's viewpoint when asked about a person's viewpoint.

Temporal decay. Long-video work documents the decay we want to decompose but does not explain it. HourVideo reports Gemini 1.5 Pro at 37.3% against 85.0% for humans. Two Causally Related Needles in a Video Haystack shows accuracy falling as the distance between cause and effect grows. IV-Bench reports prompt forgetting when visual anchors precede long video.

The closest existing work sits on either side of our question.

  • Dr.V builds a perception/temporal/cognition hallucination taxonomy and an external tool pipeline. It assumes errors are grounding failures and has no category for a prior overriding evidence that was retained. - Learning to Decode Against Compositional Hallucination (TriCD) makes the same assumption at decoding time. It amplifies spatial and temporal saliency on the premise that the visual signal was drowned out.

  • Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning is the nearest mechanistically. It shows conflict states are linearly separable in layers 15–22 of 7B models and suppresses them with probe-guided steering. However, it works on static images and long text reasoning chains.

Our contribution is neither another taxonomy nor another benchmark of aggregate accuracy. It is a measurement that splits the temporal-decay curve into loss and override, followed by a test of whether the static steering tools act on the override part only.

Research Program

Needle construction. We build three families of needles, each with a prior-consistent twin.

The spatial-relational family draws on What'sUp's tripod-controlled household scenes. Counter-prior items show configurations such as an object under a table. Their twins show the same objects in the prior-favoured configuration.

The perspective family draws on ViewSpatial-Bench scenes. Person-centric questions whose answer differs from the camera-centric answer are the counter-prior items. Questions where the two viewpoints agree are the twins.

The physical family draws on CRIPP-VQA collision clips. Items whose outcome contradicts the mass implied by appearance are paired with ones that confirm it. The CRIPP-VQA out-of-distribution sets provide extra counter-prior material.

Haystacks and depth. Each needle, rendered as a short clip, is spliced into an Ego4D-derived HourVideo haystack. We place it at five depths, measured as the time between the needle and the question: near zero, 5, 15, 30, and 60 minutes. This lets us see whether the U-shaped or cliff-shaped decay reported by MLVU and ScaleLong appears selectively on counter-prior items.

Defining the prior answer. For each item, the prior answer is whatever the model chooses when shown the question with no video. HourVideo's blind-GPT-4 condition (19.6%, near the 20% floor) is the text-only analogue of this probe.

Controlling for perception. The analysis keeps only needles the model answers correctly at depth zero. Any later error is therefore not a perception failure. This addresses the concern that the on-table prior lives in the vision encoder rather than in the language model.

Measurement. For each depth $d$, we record three per-item quantities: the error rate, the share of errors that pick the prior answer, and the outcome of a retention query. The retention query is a neutral recall question about the needle ("describe the object positions in the clip about the kitchen"). It is posed in a separate forward pass over the identical context and scored against ground truth.

The key quantity is

$$ O(d) = P(\text{main answer} = \text{prior answer},\ \text{retention correct} \mid \text{counter-prior},\ d)\; -\; P(\text{main answer} = \text{matched distractor},\ \text{retention correct} \mid \text{consistent},\ d), $$

where the second term is a false-alarm rate from the twin items. That subtraction removes generic answer noise. $O(d)$ estimates active override. The remaining counter-prior errors, those that come with failed retention, estimate passive loss.

Mechanism. On open-weight models we train linear probes on the residual stream. These models are Qwen2.5-VL / InternVL2.5-8B-class models, VideoLLaMA3-7B, and Qwen3-VL-Instruct-8B, the latter two being the models TriCD was evaluated on. Following the residual-stream study, two probes are trained at the answer position: a conflict-detection probe and a resolution probe that predicts whether the answer will follow the prior or the context. We track how both change with depth. Relapse predicts that detection stays high with depth while resolution shifts toward the prior. Pure forgetting predicts that detection itself fades.

Intervention. We apply two treatments.

  • Conflict-gated steering: a contrastive steering vector in the style of ContextFocus, built from about 1.5k contrast pairs taken from the static depth-zero condition. It is applied only when the detection probe fires, as in the long-reasoning-chain conflict paper.
  • Evidence re-injection: the needle clip is re-inserted immediately before the question.

The decomposition predicts a double dissociation. Steering should repair retained-but-overridden errors and do little for lost-evidence errors. Re-injection should do the reverse on lost-evidence errors, but should leave a residual failure on override items, because putting a counter-prior clip back into view does not stop a prior that already defeated it once.

Baselines

Do nothing. The naive reference is the unmodified model at each depth, together with the blind no-video condition that defines each model's prior. On HourVideo overall, the file reports Gemini 1.5 Pro at 37.3%, GPT-4 on captions at 25.7%, and LLaVA-NeXT at 22.3%. On perspective items, ViewSpatial-Bench reports 43.24% for InternVL2.5-8B and 34.98% for GPT-4o. What'sUp reports 56% for its best fine-tuned model. These numbers anchor how hard the needle families are before any haystack is added.

Strongest decoding-time competitor. This is TriCD from Learning to Decode Against Compositional Hallucination. It reports a 9.61% improvement on Qwen3-VL-Instruct-8B and 12.03% on VideoLLaMA3-7B on OmniVCHall. It is the strongest competitor because it is training-free, already built for spatial and temporal video hallucination, and targets the "evidence was drowned out" account directly. We run it on the same models and needle sets. Its perturbation controller gets the same number of tuning items as our steering vector (1.5k). Our method beats it if gated steering yields a larger reduction in $O(d)$ at depths of 15 minutes or more, while matching or exceeding it on overall needle accuracy. We also report where TriCD wins, which we expect to be lost-evidence items.

Strongest external-tool competitor. This is the Dr.V agent pipeline, which reports +18.60% for VideoChat2 on Dr.V-Bench. We run it on a subsample, because of its latency, and report accuracy and wall-clock time together. It is not expected to address override, since its verifier models share the same priors. Showing that is a finding in itself.

Key ablation. Ungated steering, applied at every question, isolates the contribution of conflict detection. A random-direction vector of equal norm checks that the effect is not generic activation noise. Evidence re-injection is both a baseline and one arm of the dissociation test.

Upper reference. For the perspective family, the dedicated 3B spatial model MVSM reports 82.09% on ViewSpatial-Bench. We use this to judge whether steering closes a meaningful share of the gap, not whether it matches spatially trained weights. The file reports no numbers for ContextFocus on ConFiQA or NQ-SWAP, so we have no prior value to compare against there.

Why not just…?

Why not just plot accuracy against needle depth on existing long-video benchmarks? That curve already exists in MLVU, ScaleLong, and CAUSAL2NEEDLES. A falling curve is predicted by both forgetting and relapse. Only the twin design with depth-zero filtering, the prior-directed error analysis, and the retention query can separate them. If $O(d)$ turns out flat and near zero, this reduces to the existing curve, and we would report that plainly.

Why not just assume errors revert to the blind answer because evidence is gone? This is the real threat. A model that loses the evidence will naturally fall back on the prior, which looks like relapse but is passive. The retention query exists to separate these cases. There is a caveat: a separate recall pass might succeed through a different route than the main answer uses. So we cross-check with the probe that reads whether needle content is decodable at the answer position in the main pass, and we claim active override only where the behavioral and probe measures agree.

Why not just retrieve the relevant clip and put it back in context? If most decay is passive loss, retrieval is the right fix and our steering adds little. Re-injection is therefore built into the design as a treatment. The decisive result is whether counter-prior items retain a failure after re-injection that steering removes. The multimodal conflict benchmark's finding that models detect conflicts yet still follow memory suggests such a residue exists, but it has not been shown for video.

Why not just fine-tune on spatial or physics data? What'sUp reports that spatial fine-tuning helped on in-distribution data but hurt on its controlled benchmark. STSBench's driving-tuned models (DriveMM 39.51%, OmniDrive 29.33%) scored below the general InternVL2.5-8B (46.07%). Fine-tuning also changes the priors we are trying to measure. We include a small LoRA fine-tune on the depth-zero needles as a secondary comparison, evaluated on held-out needle families.

Proposed Experiments and Decisive Checks

The first experiment establishes the phenomenon. For each model and needle family, we plot counter-prior and consistent accuracy against depth, restricted to items correct at depth zero, and decompose counter-prior errors into override and loss. The hypothesis is supported if two things hold with bootstrap confidence intervals over items: the gap between the counter-prior and consistent curves widens with depth, and $O(d)$ increases significantly from 5 to 60 minutes. The hypothesis is refuted if the curves stay parallel, or if all excess errors come with failed retention.

The second experiment is mechanistic. It asks whether the detection probe's accuracy stays near its depth-zero level while the resolution probe shifts toward the prior. Both probes are placed by layer sweep, with the reported layers 14 and 16–17 in Llama3-8B and layers 15–22 in 7B multimodal models as starting points.

The third experiment is the double dissociation. We compare gated steering, re-injection, TriCD, and their combination, measuring override errors and loss errors separately. The claim that temporal decay contains a steerable knowledge-conflict component survives only if the treatment-by-error-type interaction is significant and in the predicted direction on at least two of the three needle families.

As a transfer check, the steering vector is built from the physical family and applied to the spatial family. This tests whether "prior override in long context" is one internal direction or several family-specific ones.