Where Video Models Lose the Thread: Separating Perception from Execution Failures Under Temporal Distraction

Video language models can often say which way a ball rolled in a clean clip. Add an unrelated second object moving the other way, a few seconds of irrelevant footage, or a misleading subtitle, and they fail. Text models show the same pattern: semantically coherent but irrelevant context cuts accuracy by about 45% on average, even though the models answer correctly once the distraction is removed (Breaking Focus). Nobody has measured this for video. More importantly, nobody has asked the question that decides how to fix it. When a distractor makes a video model answer wrongly, does the model's internal state still contain the correct answer (an execution failure), or has the distractor overwritten the relevant perception (a perception failure)? The two call for different fixes. An execution failure could be repaired cheaply at inference time by nudging a few internal activations. A perception failure needs new training data or a different architecture.

We propose to answer this with code-generated video pairs that are exactly matched. An LLM writes Python scene programs in the style of SynRL. Each program renders a clean clip and a distracted clip whose correct answer is guaranteed by the program's metadata to be identical. With such pairs we can train simple readout classifiers ("probes") on the model's hidden states, copy activations from clean runs into distracted runs, and derive sparse steering directions. We then test whether a steering direction learned only on synthetic pairs repairs distraction failures on real benchmarks. The deliverable is a map of which kinds of temporal distraction are execution failures and which are perception failures, together with a head-to-head test of whether inference-time steering can compete with preference training for the execution cases.

Core Claim

We hypothesize that distraction in video models splits along a physical line. Temporally interleaved distractors, such as irrelevant events inserted before or after the target event, and narrative distractors, such as misleading subtitles, mostly cause execution failures. In these cases the correct temporal fact stays linearly decodable from mid-layer hidden states while the output goes wrong. Spatially concurrent distractors, such as a second similar object moving at the same time, mostly cause perception failures. There the model binds the motion to the wrong object, and the fact is no longer decodable. If this holds, sparse activation steering should recover most of the accuracy gap for the first family and little for the second, and preference or data-based training should be needed only for the second.

The recoverable fraction of the gap is defined as

$$R = \frac{a_{\text{fix}} - a_{\text{dist}}}{a_{\text{clean}} - a_{\text{dist}}},$$

where $a_{\text{clean}}$ and $a_{\text{dist}}$ are accuracy on matched clean and distracted clips and $a_{\text{fix}}$ is accuracy on distracted clips after a mitigation. This mirrors the gap-recovery measure in Base Models Know How to Reason, Thinking Models Learn When. That paper found that steering a base model recovers nearly 80% of the gap to an RL-trained reasoning model. That is the text-domain precedent for "the capability is present but not being used."

Positioning

The motivating evidence comes from three places. Breaking Focus shows that distraction is an execution failure in text, uses tree search to construct adversarial distractors, and finds that "ignore irrelevant info" prompts fail while DPO (Direct Preference Optimization, training the model to prefer its clean-context answer over its distracted one) works. It never looks inside the model and never touches video. TempCompass shows that video models already fall to near-chance (25–33%) on direction and speed questions, so temporal perception is fragile even without distraction. ReXTime shows that models can answer questions without localizing the evidence (GPT-4o mIoU 36.28 against a human 61.11). This is consistent with answers being assembled from the wrong segments.

The methods we reuse come from other work. Scene programs with exact metadata come from SynRL. That work deliberately uses clean backgrounds and never tests clutter, and adding clutter is our extension. Per-dimension adaptive steering comes from AUSteer, which edits at most 100 dimensions per layer and beats whole-layer steering on text tasks. Treating internal features as checkable facts is inspired by ActivationReasoning, though we use plain linear probes rather than sparse autoencoders plus a symbolic solver.

Two lines of work constrain the design. AbstRaL finds that simply scaling synthetic variations does not make GSM reasoning robust to distractors. This warns that noisy-data augmentation alone may be shallow, and we test that directly. Base Models Know How to Reason finds that SFT-distilled models (models fine-tuned to imitate a teacher's reasoning) are essentially unsteerable below 32B. DyLaR, which reasons in continuous hidden states distilled from a textual teacher and was never tested under noise, may therefore behave differently from RL-trained Video-R1. We treat this as a measured variable, not an assumption.

Research Program

The first component is a matched-pair generator. An LLM writes parameterized scene programs covering the temporal primitives in SynRL and TempCompass: direction, speed comparison, event order, collision, and container tracking. A separate distractor module adds one of four perturbations while holding the target's trajectory fixed. The first is interleaved irrelevant events. The second is concurrent look-alike objects. The third is textured or real-video backgrounds composited behind the target. The fourth is LLM-written subtitles that are coherent but irrelevant, or that assert the opposite answer. Because the answer is computed from the program rather than from a captioner, the answer-invariance check is exact. This removes the hallucination-propagation risk that affects GPT-4o-labeled corpora such as LLaVA-Video-178K. Following Breaking Focus, a search over distractor parameters (count, timing, similarity to the target, subtitle content) finds the perturbations that most reduce each model's accuracy. The adversarial set is thus adaptive, not a fixed template.

The second component is the diagnosis, run on Qwen2.5-VL-7B, Video-R1-7B, and DyLaR if its checkpoints are available. For each matched pair we train layer-wise linear probes that decode the answer variable, for example "target moved left," from the hidden states at the question tokens. We report probe accuracy on distracted inputs relative to clean ones. We also patch activations, copying clean-run hidden states into the distracted run one layer at a time, to find where the failure is introduced. A category is labeled an execution failure if probe accuracy on distracted clips stays within a few points of clean accuracy while output accuracy drops. It is labeled a perception failure if probe accuracy falls along with output accuracy.

The third component is the repair. From the clean-minus-distracted activation differences we select sparse dimensions and apply AUSteer-style per-sample scaled edits, gated to fire only when a lightweight distraction detector on the hidden states triggers. Every steering hyperparameter is tuned only on synthetic pairs. We then freeze the edit and evaluate it on real video.

Baselines

The naive reference points are the unmodified models, the same models with a system prompt telling them to ignore irrelevant events and subtitles, and the same models with step-by-step prompting. Breaking Focus reports that the ignore-prompt fails in text. DyLaR reports that textual reasoning on Qwen3-VL-4B used 1,220.7 tokens for 54.0% accuracy, against 58.2% with 18.5 tokens using latent reasoning. We rerun both on our distracted sets rather than assume those results carry over.

The strongest competitor is DPO on distracted-versus-clean pairs, the method Breaking Focus found effective. It is the most direct existing fix for distraction and uses the same matched data we use. We train it with LoRA on exactly the synthetic pairs from which the steering vectors are derived, with a matched hyperparameter-search budget. Breaking Focus reports an approximately 45% average degradation and GPT-4o-mini falling from 85.7% to 22.0% on CommonsenseQA, both in text. We found no reported result on Lacuna for any mitigation of distraction in video models. Steering "beats" DPO on a category if its recoverable fraction is statistically indistinguishable from or higher than DPO's on real distracted benchmarks, and it costs no training and no loss on clean accuracy. Losing to DPO on perception-failure categories is the predicted outcome, not a failure of the proposal.

The second trained baseline is supervised fine-tuning on SynRL-style data with our distractors mixed in, with the same number of samples as DPO. It tests whether plain data augmentation matches both methods.

The ablations isolate the key ingredient, which is exact pairing. We derive steering directions from unmatched clean and distracted clips, and from pairs whose answers were labeled by a captioning LLM instead of the program. We compare whole-layer contrastive steering against sparse steering. We also run a random-direction control of equal norm.

For absolute context on the models used, the file reports Video-R1-7B at 37.1% on VSI-Bench and DyLaR-trained Qwen2.5-VL-7B at 57.5% average accuracy across its video suite. Both were measured without distractors.

Why not just…?

Why not just fine-tune on noisy synthetic video? This is the most serious threat. If supervised fine-tuning with distractors closes the gap on every category, the steering result matters only scientifically. Two things keep it from settling the question. AbstRaL found that scaling synthetic variations did not instill robust reasoning against interferential shifts. More importantly, fine-tuning cannot tell us whether the information was ever lost. The probe results answer that regardless of which fix wins. The augmentation baseline is run at equal data volume, so if it dominates everywhere we will report it plainly.

Why not just drop irrelevant frames before answering? Extraction-before-generation works for retrieval noise in text (Ext2Gen), and a frame selector should handle interleaved distractors. It cannot remove a look-alike object moving in the same frames or a subtitle overlaid on relevant frames. We include an oracle frame-selection baseline that uses the program's ground-truth relevant frames. This gives an upper bound for any selector and shows exactly which categories selection can reach.

Why not just sample frames more densely? DVBench reports that denser sampling does not monotonically improve temporal reasoning, and more frames means more distractor frames. We include one denser-sampling condition, at twice the frame count, as a check.

Why not just use whole-layer steering? AUSteer shows that whole-layer steering is intrusive and loses to sparse edits in text. We test it anyway, because visual tokens could make the sparse subset less stable.

Proposed Experiments

We run four experiments, each tied to a falsifiable prediction.

The first maps the phenomenon. We measure clean-to-distracted accuracy drops per primitive and distractor family on the synthetic suite. We also measure them on distracted TempCompass, built by spatially and temporally concatenating irrelevant clips (TempCompass's own construction operations) and adding generated subtitles. The prediction is a drop comparable in size to text distraction, with concurrent distractors hurting most.

The second is the diagnosis. Probe accuracy should stay near clean levels for interleaved and subtitle distractors and fall for concurrent look-alikes. Patching should localize the interleaved failure to middle-to-late layers. If probes collapse for every category, the execution-failure framing is wrong for video, and the remainder of the work shifts to data-based fixes.

The third is the repair-transfer test. Steering tuned only on synthetic data is evaluated on distracted TempCompass, on ReXTime (whose untrimmed videos contain natural distractors, analyzed by the Sequential, Cause-Effect, and Means-to-an-End categories), and on NExT-QA with added subtitles. We report the recoverable fraction $R$, clean-accuracy change, and added latency against every baseline above. The decisive result is a significant difference in $R$ for steering between execution-type and perception-type categories, with DPO showing a much smaller difference. That would confirm that the diagnosis predicts which fix works.

The fourth compares model families. It tests whether RL-trained Video-R1 is more steerable than a distillation-trained latent reasoner. That result would extend the text-only finding of Base Models Know How to Reason to video.