Counterfactual Evidence Pruning for Causal Multi-Hop Video Search

When a video model has to explain why something happened, say a memorial service late in a film, it usually has to search in two steps. First it finds the effect. Then it works out who or what connects the effect to its cause (the "bridge"). Then it finds the cause, often minutes earlier. Current systems either look at everything at once or collect clips step by step and keep all of them. Both approaches fill the model's input with clips that are related to the question but did not cause the event: the same character in a different scene, an event that happened just before, an object that co-occurs. Models are easily misled by this kind of irrelevant input. On the benchmark built for this two-step causal search, the best frontier model correctly locates both cause and effect only about 9.3% of the time, while humans manage about 88% (Two Causally Related Needles in a Video Haystack).

We propose that a step-by-step video searcher should decide which retrieved clips to keep with a causal test rather than a relevance test. A clip earns its place in the working context only if two things hold. First, removing it makes the model noticeably less able to account for the effect. Second, that support depends on the clip coming before the effect, as a real cause must. Clips that look relevant but fail either test are discarded before the next search step. The idea is cheap to run at inference time on a frozen model. It makes a falsifiable prediction: it should beat relevance-based pruning specifically when the distractors are on-topic but causally inert, and not otherwise.

Core Claim

Two strands of prior work point to the same failure. Text work shows that reader accuracy falls as more retrieved material is added (Improving Zero-shot Reader by Reducing Distractions). Shrinking the context step by step and stopping once it is judged sufficient recovers that accuracy (CompAct). Video work shows that models invent causes from mere co-occurrence or adjacency in time (MECD). It also shows that models answer from scene and language shortcuts rather than from the order of events (Understanding Long Videos with Multimodal Language Models; NExT-QA).

Our claim is that these are the same failure in a multi-step search. The noise that hurts most in causal video search is not off-topic footage, which any relevance filter removes. It is on-topic footage with no causal role, and only a causal test removes it. We define the test with two quantities, both computed by the frozen video model scoring a short text description $e$ of the effect, given the current set of kept clips $E$:

$$N(s) = \log p_\theta(e \mid E) - \log p_\theta(e \mid E \setminus \{s\}), \qquad A(s) = \log p_\theta(e \mid E) - \log p_\theta(e \mid E^{\,s \to \text{after}})$$

Here $p_\theta$ is the model's probability of the effect description. $E \setminus \{s\}$ is the kept set with candidate clip $s$ removed. $E^{\,s \to \text{after}}$ is the same set with $s$ moved to after the effect in the timeline. The necessity score $N(s)$ asks whether the clip matters at all. The asymmetry score $A(s)$ asks whether it matters because it comes first. A clip whose support survives being moved after the effect is co-occurrence evidence, not causal evidence.

A candidate is kept only if both scores exceed thresholds calibrated against matched control clips. The controls are clips of the same length taken from adjacent positions in the same video. Search stops when the last kept clip is necessary and the next search step adds no clip that passes. This gives a concrete stopping rule for the "when is the chain complete" problem that iterative retrieval methods leave open.

Research Program

The searcher builds on the crop-and-inspect loop of Video-o3. In that loop, a multimodal model repeatedly requests a time window of the video at a chosen frame density. When answering, it hides its low-resolution overview of the whole video so that it must rely on the windows it fetched. Video-o3 already removes one source of noise, the global overview. However, it keeps every fetched window, caps search at 8 turns, and has no notion of causal dependence between events. We keep its loop and tool. After each fetch, we insert the counterfactual pruning step. We also replace its fixed turn cap with the stopping rule above, so long searches do not exhaust memory through accumulated clips.

The causal test borrows its logic from MECD's adaptation of Granger causality, which asks whether masking a premise event degrades prediction of the outcome. It differs from MECD in three ways. MECD trains a dedicated model over pre-segmented, captioned events. Our test runs zero-shot inside an open-ended search over raw, unsegmented video. It acts on whatever windows the agent happened to fetch. It adds the order-reversal check, which MECD does not use.

The closest pruning idea is the dynamic tree expansion in Video-Grounded Entailment Tree Reasoning. That method drops sub-statements that do not raise confidence in their parent answer hypothesis. It prunes text claims about short clips. We prune visual evidence in long videos, and our test is ablation-based and order-sensitive rather than confidence-based. Superposition Prompting scores retrieved documents independently along parallel paths before merging them. That design cannot recognize a document whose value appears only alongside others. Because our necessity score is computed relative to the current kept set, it captures exactly that joint value, at the cost of extra scoring passes.

Cost stays modest because both scores require only teacher-forced likelihoods of a short effect description, not generation. The shared prefix can be cached. Each candidate costs two extra scoring passes, plus two for its matched control.

We will run everything on Qwen2.5-VL-7B, the backbone family used by Video-o3 and by Fostering Video Reasoning via Next-Event Prediction. Thresholds are calibrated on the single-needle portion of CAUSAL2NEEDLES and frozen before any two-needle evaluation.

Beyond the method, we build a controlled decoy stress set. This set is needed because the central prediction concerns on-topic, causally inert distractors, and existing benchmarks do not separate these from off-topic noise. MECD's 1,105 multi-event videos come with pairwise causal annotations. For each annotated cause–effect pair, we can therefore designate events annotated as non-causal but sharing entities or adjacency with the effect as decoys. We then ask the searcher to locate the cause. A second variant splices decoy clips containing the bridge entity into CAUSAL2NEEDLES videos at controlled time distances from the effect. This lets us measure the distance penalty that paper reports, with and without pruning.

Baselines

The naive reference point is the frozen backbone given uniformly sampled frames of the whole video, answering in one pass. The long-context reference point is the strongest reported model on CAUSAL2NEEDLES. Gemini 1.5 Pro reaches about 9.3% on two-needle causal visual grounding (Two Causally Related Needles), and open models are reported near chance on the hardest items. These anchor what "doing nothing clever" achieves.

The strongest existing method is Video-o3's interleaved crop search with its overview-masking at answer time. It is the only cited system that already does multi-step clue seeking in long video while explicitly reducing context noise. It reports 72.1% on MLVU, 46.5% on Video-Holmes and 60.7 mIoU on Charades-STA. We found no reported result on Lacuna for Video-o3 on CAUSAL2NEEDLES, ReXTime or MECD, so we will measure it ourselves. If its trained weights are available, we run them unchanged and add our pruning step on top. If not, we reproduce its inference loop on the same backbone and label the result clearly as an untrained reproduction. All searchers get the same frame budget per fetched window, the same maximum of fetched windows, and the same threshold-tuning data, namely the single-needle split.

The ablation that isolates our key ingredient is relevance pruning. This is the same loop, but each fetched clip is kept or dropped by the model's own judgment of whether it is relevant to the question and sufficient to answer. This matches CompAct's self-evaluation with explicit complete/incomplete decisions and Superposition-style saliency scoring, transferred to video windows. It controls for "pruning helps at all." Two further ablations use necessity alone, without the order check, and order-asymmetry alone. A third replaces our model-based scores with random pruning at the same keep rate. A fourth is a caption-then-text pipeline: the video is captioned per segment and IRCoT (Interleaving Retrieval with Chain-of-Thought) is run over the captions.

Beating the baselines means the following. Our method must exceed relevance pruning on the decoy stress set by a margin larger than seed-to-seed variance, on both cause-localization accuracy and the fraction of decoys left in context. It must match or exceed Video-o3 on the unmodified CAUSAL2NEEDLES two-needle grounding task. On ReXTime it must narrow the gap between answer accuracy and localization. That gap is large: GPT-4o reaches 73.7% accuracy but only 36.28 mIoU (ReXTime).

Why not just…?

Why not just use relevance-based pruning, as CompAct does for text? This is the real threat, and we expect it to win or tie on footage that is simply off-topic. Most CAUSAL2NEEDLES distractors may be of this kind, in which case the causal test would add cost without gain on the headline number. We say so in advance. The decoy stress set exists to settle the question. Relevance judgments should keep entity-sharing, adjacent decoys, because they are relevant. Our necessity and order tests should drop them. If relevance pruning matches us on the decoy set, the causal test is unnecessary and we will report that.

Why not just feed the whole video to a long-context model? The two-needle result of about 9.3% for Gemini 1.5 Pro, on videos averaging only seven minutes, already shows that a large window does not produce causal search. CAUSAL2NEEDLES also reports that accuracy falls as cause and effect move apart. Our distance-controlled decoy splicing measures whether pruning flattens that curve where long context does not.

Why not just caption the video and run text multi-hop retrieval? This reduces the problem to settings where IRCoT and CompAct are known to work, so it is a serious competitor. However, the Entailment Tree work names its dependence on captions as a limitation because captions lose fine physical detail. CAUSAL2NEEDLES's image-description task asks about visual details found only in the target frames. We run the caption pipeline as a baseline. If it matches us on grounding but loses on image-description questions, that locates where the visual test is needed.

Why not just give Video-o3 more turns? More turns mean more kept windows. Text evidence shows that more retrieved context without pruning plateaus or hurts (CompAct). We test this directly by sweeping the turn cap for the Video-o3 loop with and without our pruning step and plotting accuracy against turns.

Decisive Experiments

The first decisive check is the decoy stress set built from MECD annotations and spliced CAUSAL2NEEDLES videos. We report cause-localization accuracy and the decoy-retention rate for our method, relevance pruning, Video-o3 and the caption pipeline. The pre-registered prediction is that our method's advantage over relevance pruning grows with the number of on-topic decoys and is near zero for off-topic distractors. A flat advantage would undercut the core claim.

The second check is unmodified CAUSAL2NEEDLES two-needle visual grounding and image description, broken down by cause–effect distance. The test is whether pruning reduces the distance penalty.

The third check is ReXTime cause–effect questions. We report both multiple-choice accuracy and mIoU. Prior references are GPT-4o's 73.7% and 36.28 mIoU, and VTimeLLM's 58.15% after fine-tuning on ReXTime's synthetic training set. The expectation is a larger gain in mIoU than in accuracy, because the method should improve where the model looks more than which option it picks.

The fourth check uses the scores alone, with no search, as a zero-shot causal discovery scorer on MECD's segmented events. We compare against MECD's trained 71.2% accuracy and GPT-4o's 65.5%, plus structural Hamming distance (the number of edge errors between predicted and true causal graphs). This tests whether $N$ and $A$ measure causal dependence at all, separately from the search they are embedded in.

As a guard against the language-prior shortcut, all multiple-choice evaluations are repeated on NExT-QA with options rewritten to be equally plausible, following the Entailment Tree procedure. On those rewritten options, standard models lose 8 to 10 points.