Where Video Models Lose Time: Joint Space-Time Attention Versus Training Signal

Video language models answer questions about clips by turning frames into tokens that a language model reads. They do well on "what is in the video" and badly on "what happened, in what order, and who did what to whom." On a benchmark of paired videos that look identical and differ only in timing or motion, the best frontier model gets under half the pairs right while humans get nearly all of them. This proposal asks a narrow question that the literature keeps blurring: does this failure come from the vision encoder's architecture, meaning whether it lets every patch at every moment attend to every other patch at every other moment, or from the training signal, which rarely forces a model to distinguish videos that share appearance and differ only in dynamics?

The core bet is that the two causes fail on different questions, and that this can be shown cleanly. Training on synthetic "minimal pairs," the same clip reversed, re-timed, or with two events swapped, should fix single-object temporal questions even in a cheap encoder that handles space and time in separate steps. Questions about interactions between entities across time, such as which agent yielded to which, or what caused a collision, should additionally require joint attention, where one token can relate object A at an early frame directly to object B at a later frame. The plan is a two-by-two controlled study, crossing attention type with training signal, combined with probes that locate where the temporal information is lost: in the encoder, in the connector to the language model, or inside the language model.

Background and the gap

Unified spatio-temporal attention flattens $N$ spatial entities over $T$ time steps into one sequence of $N \times T$ tokens and runs a single attention over all of them. Several papers motivate this design. Trafformer argues that stacked space-then-time layers cannot directly correlate node $i$ at one time with node $j$ at a later time, and reports 60-minute METR-LA error of 9.96% MAPE against 14.68% for a standard factorized Transformer. SLATE applies the same argument to dynamic graphs. LONGVIVIT shows that joint space-time attention tolerates dropping 75% of tokens during training, which makes it affordable, and reports 56.8% on EgoSchema. None of these papers tests whether joint attention closes a reasoning gap. LONGVIVIT attaches its encoder to a frozen language model through shallow adapters and evaluates standard video question answering.

The reasoning-gap literature supplies the diagnosis. TimeBlind builds minimal pairs and finds that scaling from 8B to 72B parameters, or from 8 to 32 frames, barely helps. STSBench finds that agent-to-agent interaction questions trail ego questions by 20–30 points. It also finds that a text-only model given ground-truth trajectories (57.08%) beats a vision model reading pixels (InternVL 2.5 8B, 46.07%). That is a constraint on this proposal: a structured, symbolic description of the scene can be enough, and the hard part is extracting it.

The closest existing work falls into three groups. The first is architectural: STAVEQ2 inserts temporal attention after spatial attention inside each block of the vision encoder. It is factorized rather than joint, and it reports no ablation on interaction-heavy questions. The second is training on reasoning traces: EgoThinker, VIDEOP2R and Video-DPL fine-tune with reinforcement learning on spatio-temporal chains of thought or timestamped evidence, and leave the frame encoder unchanged. The third is modular: STEMO-Track and CLEVRER's NS-DR convert video into object tracks or symbolic states and reason over those. None of these separates architecture from training signal, and none predicts which question types each one fixes. That separation is what this proposal adds.

Core Claim

Our hypothesis is that minimal-pair training is sufficient for single-entity temporal structure: direction, order, and speed of one thing. Joint attention, by contrast, is necessary for cross-entity, cross-time structure: interactions, causation, and counterfactuals. This makes a falsifiable prediction. In the two-by-two design, the joint-attention gain over factorized attention, holding training fixed, should be near zero on single-entity questions and clearly positive on interaction questions. If joint attention helps uniformly, the claim about mechanism is wrong. If it does not help on interactions, the architectural argument from Trafformer and SLATE does not transfer to video reasoning.

Research Program

All variants start from the same open video language model, Qwen2.5-VL-7B, which STAVEQ2 also uses as its reference point. The pretrained per-frame vision encoder stays fixed. On top of it we add a small temporal module of four transformer layers operating on spatially pooled patch tokens, so that $N$ stays manageable. The module comes in two versions with matched parameter counts. The factorized version alternates spatial attention within a frame and temporal attention per patch location, following STAVEQ2's design with 1D temporal and 2D spatial rotary encodings. The joint version runs one attention over all $N \times T$ tokens with the same encodings. To keep the joint version affordable, it uses LONGVIVIT's random dropping of 75% of tokens during training and no dropping at inference.

The training signal also comes in two versions. The standard version is ordinary instruction tuning on video question-answer data. The minimal-pair version adds, at no annotation cost, transformed copies of each training clip: time reversal, a speed change, a swap of two event segments, and on CLEVRER, the counterfactual video provided by the simulator. It also adds questions whose answers differ between the original and the transformed copy. For each such pair $(v, v')$ and question $q$, with answers $a \neq a'$, the loss is the usual answer likelihood on both videos plus a margin term:

$$ \mathcal{L}_{\text{pair}} = \max\big(0,\; m - [\log p(a \mid v, q) - \log p(a \mid v', q)]\big), $$

where $p$ is the model's answer probability and $m$ is a fixed margin. The term penalizes the model when it assigns the answer belonging to $v$ almost as much probability under $v'$. This is the failure TimeBlind exposes, where a single frame or shuffled frames already determine the model's answer. Clips come from SSv2 and the CLEVRER training split, and no TimeBlind videos are used. Because speed-change augmentation is close to TimeBlind's "Event Attributes" tier, we treat its "Structural Event Logic" tier as the cleaner held-out test.

The diagnostic arm uses linear probes. For each variant and for each layer of the encoder, the temporal module, and the language model, we train a linear classifier on frozen features to tell the two members of a held-out minimal pair apart. Where probe accuracy is high in the encoder but the model's answer is wrong, the information is lost downstream, in the connector or the language model. Where probe accuracy is near chance already in the encoder, the loss is architectural. Probing runs separately for single-entity and interaction pairs.

Baselines

The do-nothing reference is the unmodified Qwen2.5-VL-7B with the same evaluation prompts. It anchors every gain. The file gives its VITATECS directional accuracy as 80.0% (as reported in STAVEQ2); no reported result was found on Lacuna for it on TimeBlind or STSBench.

The strongest architectural competitor is STAVEQ2. It is the most direct published attempt to fix temporal blindness inside the encoder, and it reports 87.6% on VITATECS at 7B and 73.9% on Video-MME at 72B. We reimplement its factorized temporal attention as our factorized cell, with the same data, steps, and tuning budget as the joint cell. Where released weights exist, we also run the released model to confirm that the reimplementation matches. No reported result was found on Lacuna for STAVEQ2 on TimeBlind. Beating it means the joint-attention plus minimal-pair cell exceeds the factorized plus minimal-pair cell on interaction questions by more than the seed-to-seed spread, while matching it on VITATECS.

The strongest competitor based on reasoning-trace training is a VIDEOP2R-style setup: observation-then-think supervised fine-tuning followed by process-aware reinforcement learning. We apply it to the unmodified encoder with a compute budget equal to our full training run. We replace VIDEOP2R's proprietary judge with an open model and apply the same judge to all arms. On TimeBlind the frontier reference points are Gemini 3 Pro at 48.2% and GPT-5 at 46.3% instance accuracy, both reported by TimeBlind. These come from much larger models, and we cite them only for context.

The modular competitor is a track-to-text pipeline in the spirit of STEMO-Track. An open detector and tracker produce JSON object trajectories, which the same 7B language model reads without pixels. Its ceiling is set by STSBench's text-only GPT-4o with ground-truth trajectories, at 57.08%. On CLEVRER the modular reference is NS-DR at 42.2% on counterfactual questions.

The key ablations are the off-diagonal cells: joint attention with standard training, and factorized attention with minimal-pair training. Together they isolate each ingredient. A further ablation keeps the joint module but restricts its attention mask to same-frame or same-location pairs, recovering factorized connectivity at identical parameter count. This attributes any gain to cross-time, cross-location attention edges rather than to a difference in parameters.

Why not just…?

Why not just train with chain-of-thought and reinforcement learning on the existing encoder? EgoThinker and VIDEOP2R report gains of a few points, VIDEOP2R between 1.9% and 9.1% over Qwen2.5-VL-7B. However, TimeBlind finds frontier models that reason step by step still below 50% on minimal pairs, and STAVEQ2 reports that few-shot prompting can hurt directional temporal tasks. Reasoning text cannot recover a distinction the frame tokens never encoded. This alternative is still a real threat on single-entity questions, so it gets a full baseline arm. The probes settle the matter: if the minimal-pair distinction is decodable from the unmodified encoder's tokens, then reasoning-trace training is the right fix and our architectural claim fails.

Why not just use factorized attention, which is cheaper? Our own hypothesis predicts that factorized attention is enough for single-entity questions. The open question is interactions, where Trafformer's argument about direct cross-node, cross-time correlation should apply and STSBench locates the largest gap. The matched factorized cell and the masked-joint ablation test this directly. If they tie the joint module on STSBench agent-to-agent questions and CLEVRER explanatory and counterfactual questions, we will recommend factorized attention.

Why not just extract tracks and reason over text? STSBench shows this works well when the trajectories are correct, which makes it the most serious alternative. Its weaknesses are extraction quality, latency, and losing appearance detail, all of which STEMO-Track itself concedes. We run the pipeline with an open tracker and report both accuracy and wall-clock cost. If it matches the joint cell on interaction questions at acceptable cost, the honest conclusion is that the effort belongs in better extraction, not in encoder attention.

Why not just scale the model or sample more frames? TimeBlind reports negligible gains from 8B to 72B and from 8 to 32 frames, and EgoSchema reports that more frames sometimes hurt. We include an 8-versus-32-frame sweep on the do-nothing baseline only to confirm this in our setup.

Proposed Experiments

The evaluations are TimeBlind (instance accuracy, by tier), STSBench/STSnu (accuracy split into ego and agent-to-agent categories), CLEVRER (descriptive, explanatory, predictive, and counterfactual accuracy), and VITATECS (direction). Video-MME and EgoSchema check that general ability is not traded away. Each cell runs with three seeds under an identical token and step budget, and we report the joint-attention compute overhead alongside accuracy.

The decisive check is the interaction contrast. Define it as the gain of joint over factorized attention, holding minimal-pair training fixed, on agent-to-agent and causal questions, minus the same gain on single-entity questions. The hypothesis survives if this contrast is positive and exceeds seed variance, and if the probes show that factorized encoders fail to linearly separate interaction minimal pairs that joint encoders separate. It fails if the contrast is near zero, or if the track-to-text pipeline matches the joint cell on interactions. Either outcome tells practitioners where to spend compute: on encoder connectivity, on training signal, or on structured extraction.