Video language models are often asked what happened to an object: is the onion chopped yet, which bowl holds "the mixture," did the batter actually go into the oven. Current models often answer these questions from the recipe rather than from the frames. The answer "chopped" is right most of the time because recipes usually get followed, so the model scores well without looking. A recent result offers a cheap route to better temporal reasoning. Temporal Reasoning Transfer from Text to Video shows that the vision encoder already carries the temporal signal and that the language model is the bottleneck, so fine-tuning the language model on synthetic text-only temporal puzzles improves video performance without any new video. The danger is that the same route applied to object states will teach the model even more procedural script knowledge, making it more confident that onions end up chopped.
We propose to build text-only state-tracking puzzles in which the written plan and the observed events sometimes disagree, and the correct answer always follows the observations. Each puzzle pairs a procedure ("chop the onion, add it to the pan") with a timestamped stream of short, caption-like observations taken from real narration text. In a controlled fraction of puzzles the observations break the script: a step is skipped, fails, happens out of order, or an ingredient left idle for ten steps is later merged into a new entity. Our central claim is that text-only state training transfers to video only as a grounding skill when the training data makes the script prior unreliable. When plan and observations always agree, the same training should improve benchmark scores while leaving, or even deepening, the model's habit of answering from language instead of frames. This claim can be tested by comparing the two training sets on real videos where the script is violated.
Why this, and why now
Four observations from the literature line up. The first two motivate the proposal, the third supplies the method, and the fourth constrains it.
On the motivating side, the evidence that models answer from priors is strong. Stress Tests REVEAL builds controlled failure tests and finds that instruction tuning makes models about three times more likely to accept a false textual claim over the video. Its physics-defying clips, such as a candle that un-melts, expose a "temporal expectation bias." Its probing shows temporal signal present in early vision layers and suppressed in later language layers. TempCompass shows that Video-LLMs often fail to beat image models given a single middle frame, and collapse to 25–33% on speed and direction. Tracking the Truth shows that restricting models to one frame drops its prerequisite-checked benchmark to 28–32%. Together these say the failure sits in how the language model weighs evidence against expectation, which is exactly what text-only training can reach.
The method comes from T3 (the text-to-video transfer paper above). It generates GPT-4 temporal puzzles about order, attribute change, and temporal grounding from image captions, and fine-tunes only the language backbone, with self-attention unfrozen being more effective than MLP layers. We reuse this training recipe. T3's puzzles cover coarse, discrete relations and do not track objects that change composition, merge, or get hidden inside containers. More importantly, nothing in T3 separates learning to follow observations from learning plausible temporal scripts.
The constraint comes from the synthetic-state line of work. Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding generates before, mid, and after state text with GPT-4o-mini for WikiHow steps. What Changed and What Could Have Changed? generates state-change and misordered-step counterfactuals from EgoClip narrations with Llama 3.1. Both use this text to train video encoders contrastively. Both acknowledge hallucinated or low-probability states, and neither audits for single-frame shortcuts. The research file's Gap 1 states the open question directly: nobody has checked whether synthetic state supervision teaches visual transitions or static noun and script associations. The synthetic-to-real literature adds a second constraint on register. Models trained on fluent LLM text degrade on telegraphic human text (Lorge et al.; Ramineni et al.). Video captions and narrations are telegraphic, so puzzles written in GPT prose may not transfer.
Finally, the closest work on entity tracking in video, Predicting Implicit Arguments in Procedural Video Instructions, trains Qwen2-VL on about 18K GPT-4o silver annotations to recover omitted arguments. It is evaluated on short segments, never on videos where the procedure goes wrong. Its finding that structured tuples before free text help, together with the finding in Tracking the Truth that JSON state beats prose summaries, shapes our answer format.
Core Claim
Let $p$ be the fraction of training puzzles in which observations contradict the plan. Let $A_{\text{viol}}$ and $A_{\text{conf}}$ be a model's accuracy on real-video state questions whose correct answer violates or conforms to the procedural script. We define the evidence-following gap as
$$G = A_{\text{conf}} - A_{\text{viol}},$$
which is large when a model answers from the script and near zero when it answers from the frames. We predict three things. Training at $p=0$ (script-consistent puzzles) raises $A_{\text{conf}}$ and leaves $G$ large or larger. Training at intermediate $p$ shrinks $G$ while keeping $A_{\text{conf}}$. Observations written in real narration register transfer better than the same content paraphrased into fluent LLM prose. If $G$ does not depend on $p$, the claim is false.
Research Program
The puzzle generator. We extract entities and state-changing verbs from real procedural sources, namely YouCook2 and CrossTask step annotations and EgoClip narrations. The EgoClip strings are already telegraphic, of the form "C cuts onion." A small symbolic simulator tracks each entity's state and handles four kinds of event: transformation (whole to sliced to cooked), merging (onion plus oil becomes "the base"), containment (the garlic goes into the jar), and idleness over long spans. The last category deliberately covers the non-Markovian recombinations that CAST notes its immediate-predecessor residual cannot handle.
Each episode has three parts: a plan, an observation stream with timestamps, and questions. The questions ask what state an entity is in at time $t$, what "the mixture" contains, where an entity is, and whether a step actually happened. The simulator computes answers from the observations only. Violations come from real error types rather than LLM imagination: skipped steps, failed steps, reordered steps, and non-events such as the glass being caught rather than dropped. Because violations are made by editing a symbolic trace, they avoid the hallucinated physical scenarios that affect LLM-written counterfactuals. Answers are emitted as a short structured state record followed by the text answer.
Training. Following T3, we fine-tune LongVA-7B, the backbone on which T3 reports results, updating only self-attention on text. We repeat the key runs on Qwen2-VL-7B to test whether the effect generalizes across backbones. We sweep $p \in \{0, 0.25, 0.5\}$ at a fixed number of tokens. We also train a register ablation in which the identical observation streams are paraphrased by an LLM into fluent sentences.
The real-video violation test set. No existing benchmark isolates script-violating state questions, so we build one from real footage in three ways. First, EgoPER contains real procedural mistakes, and we convert its error annotations into state questions whose correct answer contradicts the recipe. Second, for YouCook2 and CrossTask clips we pair forward and time-reversed versions and ask end-state questions. A script-following model gives the same answer to both, while an evidence-following model flips. Third, we shuffle frames, and a model that treats video as an unordered bag of images, as frame-shuffling studies (Ye et al.; Bagad et al.) found for many models, should be unaffected. Following the filtering practice of VideoEval-Pro, we drop any item that a single-frame or language-only baseline solves, and we report accuracy with the single-frame and language-only baselines alongside.
Baselines
The naive reference point is the unmodified LongVA-7B. It sets $A_{\text{conf}}$, $A_{\text{viol}}$, and the single-frame gap before any intervention, and tells us how much of each benchmark is solvable from priors.
The strongest existing text-only method is T3 itself, retrained with its own puzzle mix at our token budget on the same backbone and with the same self-attention-only updates. Its reported number is a rise on the Video-MME Temporal Reasoning subtask from 37.3 to 49.7 on LongVA-7B (Temporal Reasoning Transfer from Text to Video). For TempCompass, the file reports only that T3-trained LongVA-7B beat ShareGPT4Video-8B, with no aspect-level figure; we will measure the Attribute Change and Event Order aspects ourselves. Beating T3 means matching its Video-MME temporal score within noise while cutting $G$ on the violation set by a margin larger than seed variance across three seeds.
The key ablation is the script-consistent generator at $p=0$. It uses the same entities, simulator, register, token count, and hyperparameter search, and the only difference is that observations never contradict the plan. This isolates the proposal's one ingredient. If it matches the $p>0$ models on $A_{\text{viol}}$, the idea is wrong.
The strongest real-video competitor is encoder- or model-level training on LLM-generated state and counterfactual text in the style of What Changed and What Could Have Changed?. We run it as LoRA fine-tuning of the same backbone on EgoClip clips paired with Llama-generated before/after and counterfactual descriptions, at matched GPU-hours. Beating it means lower $G$ at equal or lower compute. Tying it would still favor the text route on cost, but would not support the claim that text-only training grounds better.
As external reference points on existing benchmarks, we report against the structured pipelines rather than expecting to surpass them. On STEMO-Bench, STEMO-Track with Gemini-3-Flash reports 77.1% target accuracy and 81.3% consistency, against 61.4% and 69.2% for Gemini-3.1-Pro (Tracking the Truth). On HSR-Bench, StateTrace lifts VideoLLaMA3-7B from 39.6% to 64.2% (StateTrace). On Implicit-VidSRL, iSRL-Qwen2-VL reports a 17% relative F1 gain on implicit what arguments and 14.7% on where/with over GPT-4o (Predicting Implicit Arguments in Procedural Video Instructions). Our Qwen2-VL run is fine-tuned from the same base for comparison. Closing a meaningful share of the HSR-Bench gap with no tracker at inference would be a strong result.
Why not just…?
Why not just scale the model and feed more frames? The file's evidence argues against this. On STEMO-Bench, the larger Gemini-3.1-Pro trails a structured pipeline on a smaller model by about 16 points. In REVEAL, Gemini 2.5 Pro falls from 74% to 8% when information is split across frames. TempCompass authors conclude that more frames or parameters is insufficient. Our reversal-pair test will show directly whether a larger baseline flips its answer when the video is reversed.
Why not just train on real video with LLM-written state descriptions, as TSS and the state-change counterfactual work do? This is the real threat, and it is why we include a compute-matched baseline of this kind. Two arguments favor our route. T3's probes place the bottleneck in the language model, not the encoder. These methods also inherit hallucinated states and are never audited for single-frame shortcuts. Still, real-video training could reduce $G$ just as well, and the matched comparison will settle that.
Why not just use T3, or any state puzzles, without the script-breaking? This is exactly the $p=0$ ablation. Our hypothesis is that consistent puzzles improve headline scores through stronger scripts. If $G$ does not depend on $p$, we drop the claim.
Why not just use an explicit tracker like StateTrace or STEMO-Track? These pipelines work, but they add offline multi-stage latency. They also fail in cascades when SAM2 drops an object (StateTrace). And they do not model compositional changes such as dissolving or merging. We treat them as upper references. Our goal is to move a plain end-to-end model toward them without adding any tracker at inference time.
Proposed Experiments and Decisive Checks
The first decisive check is the dose-response of $G$ on the real-video violation set as $p$ rises from 0 to 0.5, with $A_{\text{conf}}$ held within noise. The second is the reversal flip rate, the fraction of forward/reversed pairs where the answer correctly changes. We expect it to rise for $p>0$ and to stay near the base model's level for $p=0$. The third is the single-frame gap on TempCompass Attribute Change and on REVEAL's temporal-expectation and sycophancy tests. A model that has learned to use evidence should lose more accuracy when reduced to one frame, not less. The fourth is the register ablation: telegraphic versus paraphrased observations, evaluated on Implicit-VidSRL and EgoPER, whose text is organic. The fifth checks for regressions on Video-MME, STEMO-Bench, and HSR-Bench, to make sure grounding gains do not cost general ability.
The result that would most change our mind is $p=0$ and $p=0.5$ producing the same $G$. That outcome would mean text-only training shifts which scripts a model holds, not whether it looks at the frames.
References
- Temporal Reasoning Transfer from Text to Video
- Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
- TempCompass: Do Video LLMs Really Understand Videos?
- Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models
- Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding
- What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
- Detecting the clinical features of difficult-to-treat depression using synthetic data from large language models
- Towards Understanding Bias in Synthetic Data for Evaluation
- Predicting Implicit Arguments in Procedural Video Instructions
- CAST: Modeling Visual State Transitions for Consistent Video Retrieval
- HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
- Test of Time: Instilling Video-Language Models with a Sense of Time
- VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
- StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videos