Temporal Lures: Training Open Video Models to Ignore Look-Alike Events

Open-weights video language models (models that read sampled video frames and answer questions in text) look competitive with proprietary systems on standard benchmarks. The gap reopens once the shortcuts are removed. When questions answerable from a single frame are filtered out and answers must be written rather than picked, open models lose more than 25% (VideoEval-Pro). Open models also show strong positional bias that Gemini 2.5 Pro largely avoids (Video-LevelGauge).

We propose that one specific and fixable cause of this gap is temporal lures. A temporal lure is an event elsewhere in the video that looks like the evidence but is not: the same action by a different person, the same object at a different time, a near-repeat of the queried moment. Text models are known to collapse under this kind of semantically coherent distraction. Accuracy fell about 45% on average in Breaking Focus, even though the models answer correctly once the distraction is removed. The strongest open video method treats irrelevant frames as filler it can skim past. A lure is built to survive skimming.

We will build training and test data in which every question comes paired with a verified lure span. We will then train an open 7B model with a reward that pays for finding the true evidence and charges for grounding on the lure. Humans are used for one narrow task that machines do poorly and cheaply checkable cases expose: confirming that the lure really does not answer the question.

Core Claim

The claim is that much of the remaining open-versus-closed gap on shortcut-free long-video questions comes from lure susceptibility, not missing perception. A span-grounded open model will often lock onto a look-alike event. Training against verified lures should reduce this, and it should transfer to natural lures in benchmarks that were never constructed adversarially.

What is new is the combination of three things. First, distraction is semantically coherent and temporal rather than random filler or static probes. Second, supervision explicitly marks where the wrong evidence is, not only where the right evidence is. Third, human verification is concentrated on answer uniqueness. That is where both automated generation and automated judging are least trustworthy.

Relation to Prior Work

Three papers motivate the proposal by documenting the gap without curing it.

VideoEval-Pro removes single-frame shortcuts and shows the open-model drop. It is a benchmark and offers no training fix.

Video-LevelGauge inserts standardized probe clips at different positions. It finds head and neighbor preference in open models that worsens with context length. Its probes are not semantically related to the surrounding events, so it measures position, not lure confusion.

Breaking Focus does three things in text only. It formalizes contextual distraction as an ability deficit, generates distractors by error-guided tree search, and mitigates them with preference training on distracted versus clean pairs. Our proposal is the temporal analogue, but it differs in one essential way. A video lure has a location, and that location can be supervised directly.

Two papers supply the method we build on, and one constrains it.

Video-in-the-Loop (ViTL) works in two stages. It skims low-frame-rate video to predict evidence spans, zooms into them, and trains Qwen2.5-VL-7B with group-relative policy optimization, a reinforcement learning method that scores several sampled answers against each other. Its reward combines temporal overlap and answer correctness. We reuse this architecture and reward structure. We reject its assumption that non-evidence frames are uninformative background.

Open-o3-Video shows that timestamp-tagged reasoning traces and a reward gated on correct timestamps prevent grounding collapse. We borrow that gating idea. Its data is filtered Gemini output and it never tests against distractors.

SynRL is the constraint. It shows that proprietary annotators make systematic temporal errors, such as reversed directions and misjudged speeds. Pipelines that trust them, including LLaVA-Video and ViTL's GPT-4o-generated VGrounding-QA, therefore inject flawed temporal labels. SynRL's remedy is to give up on real video and use programmatic shapes. Ours is to keep real video and verify the one property that matters for lures.

For the human role we rely on the author-reviewer finding. Reviewing model-drafted data was about 13 times more efficient than writing it from scratch in Bae et al.. Model-draft-then-human-verify pipelines like VisIT-Bench and DVBench follow the same pattern. DVBench also warns that expert verification of whole videos does not scale. Our design avoids that by never asking humans to watch whole videos.

Research Program

Mining lures. We mine lures in two ways, and the distinction is central to the evaluation.

Natural lures come from within a single video. We take question-answer items with gold evidence spans, starting from VGrounding-QA's 5,240 span-grounded questions and Charades-STA's grounding annotations. We then search the same video for other spans whose caption embedding or action label closely matches the evidence but whose content would yield a different answer, for example a different actor, count, outcome, or order.

Spliced lures insert a look-alike clip from another video at a controlled position. This gives exact ground truth about where the lure is. The risk is that the model learns to detect cut boundaries. To counter this, every spliced item is matched with a control in which a non-lure clip is spliced with identical boundary processing.

Verification. An open model drafts, for each candidate pair, the answer the question would receive if one watched only the lure span. Two automated judges then rate whether the lure is genuinely wrong and the true span genuinely sufficient. Humans review only short clips. They see the true span and the lure span side by side, answer the question on each, and flag ambiguity. Every item where the judges disagree goes to humans, plus a random audit sample of items where they agree. This audit lets us estimate the error rate of automatic acceptance and decide empirically how much human review is worth.

Training. The trained model emits a predicted evidence span $\hat s$ and an answer $\hat a$. Its reward is

$$ r = \mathbb{1}[\hat a = a] + \lambda\,\mathrm{IoU}(\hat s, s^{*}) - \mu\,\mathrm{Ov}(\hat s, s_{\ell}), $$

where $a$ is the gold answer, $s^{*}$ is the gold evidence span, $s_{\ell}$ is the verified lure span, $\mathrm{IoU}$ is temporal intersection-over-union, $\mathrm{Ov}$ is the fraction of the predicted span lying inside the lure, and $\lambda, \mu \ge 0$ are weights. Following Open-o3-Video, the answer term is paid only when $\mathrm{IoU}(\hat s, s^{*})$ exceeds a threshold. This keeps the model from earning answer reward from a lucky guess made while looking at the lure. The penalty term is the key ingredient, and we ablate it by setting $\mu=0$.

Evaluation sets. We release a held-out test suite. It contains a natural-lure split and a spliced-lure split, each with matched clean versions of every item, so lure sensitivity can be measured per question as the drop from clean to lured.

Baselines

The naive reference is base Qwen2.5-VL-7B with uniform frame sampling at the same frame budget. It shows how bad lure confusion is with no intervention.

A second reference is the same model with a system prompt telling it to ignore look-alike events and attend only to the queried moment. This is included because text results say it fails, and we want to confirm that in video rather than assume it.

The strongest existing competitor is ViTL. It is trained on exactly the question we care about: picking the right span in long video. Video-in-the-Loop reports 47.4% accuracy on LVBench with a 7B Qwen2.5-VL backbone, against 27.0% for GPT-4o and 33.1% for Gemini-1.5-Pro. It also reports a zero-shot Charades-STA mean IoU improvement from 42.3% to 54.0%. For reference, the top proprietary model on LVBench is Gemini-2.5-Pro at 67.4%. We will retrain ViTL ourselves on the same base questions, the same backbone, the same number of policy-optimization steps, and the same frame budget for both stages. The only difference from our method will then be the lure data and the lure penalty.

Beating ViTL means the following. On our lured test splits, the clean-to-lured accuracy drop must shrink substantially. On LVBench, our model must stay at or above our reproduced ViTL number. On VideoEval-Pro, it must improve, where no per-model score is reported on Lacuna, so we will measure both systems ourselves with the same judge.

A third baseline ports Breaking Focus to video. It applies direct preference optimization, which trains the model to prefer one response over another, on pairs of correct answers given lured input versus answers that follow the lure. Same data, same compute. This tests whether the span-level penalty adds anything beyond preference training on outcomes.

Ablations isolate the ingredients. Setting $\mu=0$ keeps the lure data but drops the penalty. Replacing lures with random non-semantic filler at the same positions tests whether semantic coherence matters, or whether any padding helps the way Video-LevelGauge's probes would suggest. Training on lures that pass only automated judges, with no human review, isolates the contribution of the human stage.

Why not just…?

Why not just use ViTL's skim-then-zoom as is? Skimming discards frames that look irrelevant at low resolution. A lure is chosen precisely because it looks relevant at low resolution, so the skim stage should be the first place ViTL fails. This is an empirical question and the first experiment answers it. If reproduced ViTL loses little accuracy on the natural-lure split, our premise is wrong and the project should stop there.

Why not just apply Breaking Focus-style preference training on distracted versus clean videos? It is cheaper and is the most serious alternative. It supervises only the final answer, though. A model can learn to output the right answer on training lures while still attending to the wrong span, which is the same shortcut behavior VideoEval-Pro exposes. The DPO baseline under matched compute settles this. If it closes the lured-split drop as well as our reward does and transfers equally to natural lures, then span-level supervision is unnecessary and we will report that.

Why not just fully automate lure verification, since splicing gives ground truth by construction? Splicing tells you where the lure is. It does not tell you whether the lure happens to also answer the question. That is a content judgment, and SynRL documents systematic errors in exactly the temporal judgments involved: direction, speed, and order. This could still be a real threat, because modern judges may be good enough. The audit sample measures the acceptance error rate of judge-only verification directly. The judge-only training ablation then measures whether that error rate matters downstream. If neither shows a difference, the honest conclusion is that humans are only needed for the evaluation set, not the training set.

Why not just sample more frames? VideoEval-Pro reports accuracy rising from 1 to 512 frames, but denser sampling also includes more lures. Video-LevelGauge reports that positional bias grows with context length, and ViTL reports an 8.6% gain from using 50% fewer, targeted frames. We test this alternative directly by evaluating the base model on the lured splits at several frame budgets. If density alone shrinks the lure drop, the problem is bandwidth, not distraction.

Proposed Experiments and Decisive Checks

The first check is diagnostic. We measure the clean-to-lured accuracy drop and the rate of grounding on the lure for base Qwen2.5-VL-7B, reproduced ViTL, and any proprietary models we can query, on both lure splits. The proposal predicts three things: open models drop sharply, ViTL's skim stage frequently selects the lure, and the drop is smaller for the strongest proprietary model. This would link lure susceptibility to the open-source gap. If open and closed models drop equally, lures are a general weakness rather than a gap explanation, and we would reframe the work accordingly.

The second check is transfer. Models trained only on spliced lures must improve on the natural-lure split, which contains no splice boundaries. This is the guard against the model simply learning to detect cuts. We pair it with the matched-control evaluation: a model that has learned cut detection will also treat spliced non-lure clips as suspicious.

The third check is the external benchmarks: VideoEval-Pro, LVBench, the Video-LevelGauge position curves, and Charades-STA mean IoU. Here we require that resistance to lures does not cost accuracy on ordinary questions. The specific prediction is that the middle-of-sequence dip in Video-LevelGauge flattens.

The fourth check is cost. We report human minutes per accepted item and the audited error rate of judge-only acceptance, so the human stage is justified by measured label quality rather than assumed.