Watching or Reciting? Counterfactual Step Order as a Test of Video-Language Faithfulness

When a video-language model (a large language model that also reads video frames) watches someone make a sandwich and writes a step-by-step summary, it has two possible sources for that summary. It can report what the video shows, or it can recite the familiar script for "making a sandwich" that it absorbed from text during pre-training. On ordinary instructional videos the two sources give the same answer, because real people mostly follow the canonical order. Current benchmarks and metrics therefore cannot tell a model that watches from one that recites. This proposal builds videos in which the two sources disagree, by splicing real annotated step segments into non-canonical orders. With these videos we measure how often models report the script instead of the footage, both in the step order they write and in the timestamps they give.

The central bet is that the "familiarity paradox" found in text summarization carries over to time in video. In text, models invent more unsupported content on familiar topics than on unfamiliar ones (Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains). If the same holds here, video models should misreport step order and pull timestamps toward canonical positions most often on the everyday procedures they appear to understand best. A related bet concerns evaluation. A "script-recital" system, meaning a text-only model that never sees the video and is given only the task name, may score close to real video models on standard instructional-video metrics. That would be the temporal counterpart of the do-nothing copy baseline that beats real systems in text simplification.

Core Claim

Standard outcome metrics for procedural video understanding, such as step-localization mAP, temporal intersection-over-union (IoU, the overlap between predicted and true time spans) and n-gram caption overlap, reward a model's prior about procedures as much as its perception, because in natural data the two are perfectly correlated. Breaking that correlation with spliced counterfactual-order videos, matched to spliced controls that keep the canonical order, will show three things. First, generative video models override shown order with canonical order at a rate well above chance. Second, that override rate grows with how strongly the model's own text-only prior favors the canonical order. Third, existing temporal-hallucination fixes reduce this override only partly, because most of them target temporal ambiguity in the visual input rather than a language prior that actively contradicts the video. The new element is not a new detector or decoder. It is a controlled measurement that separates watching from reciting in open-ended procedural output, with prior strength as the independent variable.

What Prior Work Shows and Leaves Open

Two papers motivate the hypothesis directly. Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains found inconsistency rates of roughly 15–22% on news summaries against 2–5% on PubMed, and attributed the gap to stronger priors on familiar topics. That study is text-only and treats familiarity as a domain label. We instead measure prior strength per task inside the model, and we ask about order and time. Can Shuffling Video Benefit Temporal Bias Problem built "pseudo videos" by moving target moments to random positions. It showed that biased grounding networks keep their scores when test videos are shuffled, which means they rely on dataset position priors. We reuse its splicing idea as a method. Our setting differs in three ways: we study generative video models, the prior in question is a semantic script learned from text rather than a dataset position statistic, and the output is open-ended procedure text rather than a single span.

The video-hallucination benchmarks constrain what counts as new. VidHalluc shows that models fall back on language priors when distinguishing visually similar actions, and TempCompass shows near-random accuracy on direction and speed. Both use yes/no or multiple-choice formats on short clips, and neither varies how strongly the prior favors one answer. SEASON targets temporally inverted claims such as slicing a cake before baking it. It builds a negative by averaging frame features, which erases order, and contrasts against that negative during decoding. Its authors leave open whether models default to priors when video contradicts common procedures, and that is exactly our question. Dr.V diagnoses video hallucination with a chain of external grounding tools, and Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering steers internal activations using attention heads that act as hallucination sensors. Both are evaluated on QA-style benchmarks.

On the evaluation side, the text-simplification literature supplies the lens. Trivial copy outputs beat real simplifications under overlap metrics (LENS), and those metrics are blind to factual errors (Evaluating Factuality in Text Simplification; APPLS). Invert4TVG found the video version of this problem: optimizing IoU alone degrades action understanding. We ask the complementary question, namely whether a system with no visual access at all can score respectably on these metrics.

Research Program

The source data is COIN, which has 180 tasks across 12 domains with step segments annotated against a fixed step vocabulary per task. The fixed vocabulary lets us score generated step order without an LLM judge.

For each test video with at least four annotated steps, we build three variants. The first is the original video. The second is a spliced-canonical control: step segments cut out and rejoined in their original order, with the same number of cuts and the same short neutral transitions. The third is a spliced-counterfactual video: the same segments rejoined after swapping one or more pairs of steps. The splicing control is essential. Without it, any drop on counterfactual videos could reflect editing artifacts rather than priors.

We tag each swap by whether it is physically coherent, meaning the two steps do not depend on each other, or physically incoherent, as in a step performed on an object that does not yet exist. This tag lets us check whether override is concentrated in cases where the footage itself looks odd. For every spliced video, the true step order and the true step boundaries are known exactly.

Prior strength is measured inside each model. Given only the task name and its step vocabulary, the model's own language backbone ranks or generates the step order. For each swapped pair $(a,b)$ we record $s_{ab}$, the fraction of $K$ sampled text-only scripts that place $a$ before $b$ in canonical order. Values of $s_{ab}$ near 1 indicate a strong prior.

The video model is then asked for a timestamped step-by-step summary of each variant. Its output steps are mapped to the step vocabulary, either by constrained generation or by nearest-label matching, which we validate on a hand-checked sample. The main quantity is the override rate on swapped pairs, written $O = \Pr[\text{output orders } (a,b) \text{ canonically} \mid \text{video shows } (b,a)]$. We report $O$ binned by $s_{ab}$. The familiarity-paradox prediction is that $O$ rises with $s_{ab}$ while order accuracy on spliced-canonical videos stays flat or improves with $s_{ab}$.

For timestamps, we measure boundary pull: for a swapped step, we compare whether the predicted span lies closer to its true spliced position or to the position it would occupy under canonical order. As a secondary check, we transfer the same construction to ActivityNet Captions, where events are less script-like and we expect smaller effects.

Baselines

The naive reference point is the script-recital system. The language backbone of each video model is given the task name, never the video. It outputs the canonical script, and we assign boundaries using the mean relative position of each step in COIN training data. We score it on original COIN test videos with the usual step-localization mAP and caption overlap, and compare it with the full video models. Across the file, the only reported COIN localization figure is for SSN-Fusion, which improves from 8.12% to 9.05% [email protected] with its task-consistency module (COIN). Low absolute scores like these leave room for a prior-only system to be competitive. If script-recital comes within a small margin of the video models on original videos but collapses on counterfactual ones, that confirms the evaluation-bias claim.

The base models are Qwen2.5-VL-7B and LLaVA-OV-7B, the models SEASON was evaluated on. Both are run with identical prompts, frame sampling and greedy decoding across all three video variants.

The strongest existing mitigation for order hallucination is SEASON, which reports +5.3% on VidHalluc for Qwen2.5-VL-7B over the base model and gains of up to +24.5% on TempCompass's temporal-sequence subtask. We run its released procedure with its default settings on the same models and frames. Following its paper, we accept its roughly 1.5× latency. Two further mitigations are run under the same frame budget, with any hyperparameters tuned on a held-out split of spliced-canonical videos only. These are frame numbering in the style of NumPro (Number it), which lifted Qwen2-VL-7B mIoU on ActivityNet from 12.5% to 31.3%, and temporal-aware activation engineering (Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering).

The key ablation is prior-contrast decoding. It uses the text-only, task-name-only pass as the negative in place of SEASON's frame-averaged negative, and is otherwise a matched two-pass contrastive decoder. This isolates whether the problem is a missing sense of temporal order, which SEASON's negative addresses, or an active script prior, which only the text-only negative addresses.

"Beating" a baseline here means a lower override rate in the high-$s_{ab}$ bins of counterfactual videos, with order accuracy on spliced-canonical videos dropping by no more than one point. Because the core contribution is the measurement, a negative result is equally informative. If SEASON already removes the override, the paper reports that its homogenized negative suffices.

Why not just…?

Why not just evaluate on TempCompass or VidHalluc event-order questions? Those benchmarks pose short-clip multiple-choice questions and do not vary how strongly the model's prior favors an answer, so they cannot test whether errors scale with familiarity. They also do not score open-ended timestamped procedure text, which is where an outcome metric can reward recitation. We still report their event-order subtasks to check that our override rate correlates with existing order scores.

Why not just use an LLM judge to score the generated procedures? A judge would bring its own script prior to the comparison. TempCompass and VidHalluc show that even GPT-4o trails humans on temporal reasoning by wide margins. Because the splices give exact ground-truth order and boundaries, and COIN gives a closed step vocabulary, scoring needs no judge. We use humans only to audit the label-matching step.

Why not just add frame numbers, as NumPro does? Frame numbers address the reporting gap between where the model attends and the timestamp it writes. They do nothing to stop the model from naming the wrong step at a correctly read frame. Our design measures that difference directly: if NumPro improves boundary accuracy on spliced-canonical videos but leaves order override on counterfactual videos unchanged, the two failures are separable.

Why not just use SEASON's contrastive decoding? This is the real threat. Its frame-averaged negative keeps the video's content but erases its order, so the negative pass may itself fall back on the script, and contrasting against it could suppress prior-driven tokens. The matched prior-contrast ablation settles the question. If SEASON's negative and the text-only negative perform equally, the mitigation question is closed and our contribution reduces to the diagnostic and the evaluation-bias finding. If the text-only negative wins specifically in high-$s_{ab}$ bins, the failure is a script prior and not missing order perception.

Why not just argue the splices are unnatural, so models are right to distrust them? The spliced-canonical control has the same cuts. Any override beyond the control's error rate therefore reflects order, not editing. The coherent-versus-incoherent swap tag tests whether models override even when the counterfactual footage is physically plausible.

Decisive Experiments

The first check concerns evaluation. We compute script-recital scores on original COIN videos under [email protected] and caption overlap. The evaluation-bias claim holds if recital comes within a few points of the base video models there, while scoring near zero order accuracy on counterfactual videos where the video models do not.

The second check is the familiarity paradox. We take the override rate $O$ on counterfactual videos minus the order error on spliced-canonical videos, plotted against $s_{ab}$. The hypothesis predicts a monotone increase in this difference with $s_{ab}$, together with matching boundary pull toward canonical positions. A flat curve falsifies the transfer of the text-domain paradox to video time.

The third check is the mitigation comparison: SEASON against prior-contrast decoding against activation engineering against frame numbering, restricted to the highest-$s_{ab}$ tercile and to physically coherent swaps. This tells us which kind of intervention addresses which failure, and whether any existing method already makes video models watch rather than recite.