Video-language models answer questions about video by feeding a large language model a stream of image-patch tokens, one grid per sampled frame. Nothing in that stream directly says how things moved. Motion is a relation between frames: which patch went where, and whether the whole scene shifted because the camera panned or because an object crossed it. The language model has to infer that relation inside its own layers, and the evidence says it mostly fails. On camera-motion identification, current models score 10–20% against 98% for humans. Linear probes find motion signals in early vision layers that the later language layers suppress (Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models). We read this as a specific, fixable case of the cross-modal representation gap: the visual side of the gap has no token for motion, so motion never reaches the text side in a form language can use.
We propose to compute motion explicitly before the language model sees anything. The model compares patch features between nearby frames to estimate where each patch moved. It then splits that displacement field into a global part, explained by a small camera model (pan, tilt, zoom), and a residual part that is object motion. Each part is handed to the language model as a few dedicated "motion tokens," separate from the appearance tokens. Because the comparison happens densely across frames, the model gets the cross-frame integration that REVEAL shows is missing. Because motion arrives as its own token type, appearance and motion stay disentangled, which is the property that MASH-VLM argues prevents action-scene hallucination.
Core Claim
The field holds two positions that seem to conflict. MASH-VLM masks attention so that spatial and temporal tokens never interact directly and must communicate only through text tokens. It reports gains on hallucination benchmarks. REVEAL shows the opposite pressure: when evidence is split across frames, models that cannot compare tokens across time collapse. Gemini 2.5 Pro drops from 74% to 8% on its Cross-Frame Integration test. Our claim is that the conflict is about where integration happens, not whether it happens. Do the dense cross-frame comparison in a lightweight correspondence module outside the language model. Compress the result into a few structured tokens whose content is close to how language describes motion: a camera move, plus a handful of object trajectories. The language model can then keep appearance and motion disentangled and still receive integrated temporal evidence.
The cross-modal argument is concrete. Language describes motion with a small vocabulary ("pans left," "zooms in," "moves toward the camera") that maps onto low-dimensional geometric quantities. Patch grids carry appearance, and motion appears in them only as a difference between grids. Instance-level contrastive training stretches points across the modality gap but does not align the underlying structure (Learning Robust Visual-Semantic Embeddings), and a single projection layer cannot serve both fine-grained and cross-modal constraints at once (Extending Multi-modal Contrastive Representations). We therefore expect a representation that already has the geometry of motion language to be easier to align than raw patches. That expectation is tested, not assumed.
Research Program
The base model is a Qwen-VL-family video model. REVEAL's mechanistic probes were run on Qwen-VL, so we can measure whether motion information survives into late layers before and after the change. The vision encoder stays frozen. For frames $t$ and $t+k$, the correspondence module estimates a displacement for each patch location $x$:
$$d(x) = \sum_{y \in \mathcal{W}(x)} \mathrm{softmax}_y\!\left(\frac{\langle f_t(x), f_{t+k}(y)\rangle}{\tau}\right)(y - x).$$
Here $f_t(x)$ is the frozen encoder feature at patch $x$ in frame $t$, $\mathcal{W}(x)$ is a local search window, and $\tau$ is a temperature. The window keeps the cost at $\mathcal{O}(N\,|\mathcal{W}|)$ per frame pair rather than $\mathcal{O}(N^2)$. This is the same local cross-frame correlation used in shifted-window video attention (Zha et al., Jin et al.). Its output, however, is a motion field rather than updated features.
Camera motion comes from a weighted fit of a global affine model:
$$(A, b) = \arg\min_{A,b} \sum_x w(x)\,\lVert d(x) - (Ax + b)\rVert^2,$$
where $w(x)$ is the confidence of the correspondence softmax. The six parameters plus fit quality form a camera token. The residual field $d(x) - (Ax+b)$ is grouped into $K$ clusters of coherent object motion, with $K$ around 4. The grouping uses the adaptive clustering idea of VQToken. Each cluster becomes an object-motion token that holds its mean residual displacement, its spatial extent, and a pooled appearance vector saying what moved. Small connectors map these vectors into the language model's embedding space. Because the correspondence module is cheap, it can run on adjacent frames at a higher frame rate than the language model's appearance sampling. This targets the coarse 32-frame sampling that GroundFormer names as a failure source for fast, subtle motion.
Training has two stages. First, only the motion connectors are trained, with the language model frozen, on clips with synthetic camera trajectories and templated motion captions. The clips are drawn from Ego4D and Charades sources that are disjoint from every evaluation split. Second, the full model receives standard video instruction tuning with the same data mixture used for every baseline. Inside the language model we test two attention layouts: unrestricted attention, and a MASH-style mask in which motion and appearance tokens communicate only through text. This directly tests whether disentanglement still helps once integration has been moved outside the language model.
Baselines
The naive reference point is the unmodified base model, after the same stage-two instruction tuning on the same data. Every gain must clear this model, because instruction-tuning data alone shifts behavior. REVEAL reports that instruction tuning makes models roughly 3x more sycophantic.
The strongest existing architectural competitor is MASH-VLM. It adds separate temporal tokens, DST-Attention masking and Harmonic-RoPE positional IDs to a Video-LLM, and it is the closest prior attempt to give temporal information its own channel. Its reported results are 57.6% average on MVBench and 57.85% average on UNSCENE binary tasks. We will reimplement its two components on our base model with the same data and an equal tuning budget, so that the comparison isolates the token design rather than the backbone. Beating it means matching or exceeding it on UNSCENE, which is its home ground, while clearly exceeding it on REVEAL camera-motion and cross-frame integration. MASH-VLM's temporal tokens are pooled appearance features, so we predict it gains little on those two tests.
The positional-encoding competitor is VideoRoPE, which uses 3D rotary positions with low-frequency temporal allocation. It is relevant because it is the cheapest way to make temporal order easier for the language model to read. Its reported result is a +12.4% gain over M-RoPE on V-NIAH and V-NIAH-D. We swap it into the base model under the same training budget.
For object-centric reasoning, Chain-of-Glimpse reports 83.3% on NExTQA. It is a search-based method that tracks objects without modelling the camera, and we compare against its reported number on NExTQA rather than retraining it.
Three ablations isolate the key ingredient. The first removes the camera/object decomposition and feeds raw pooled displacement fields. The second feeds the same displacement information but adds it into the appearance tokens instead of using separate tokens. The third keeps the motion tokens but randomizes correspondences, which preserves the token count and checks that any gain is not just extra capacity.
Why not just…?
Why not just fine-tune the existing model on the same motion data with LoRA or LayerNorm tuning? This is the most serious threat. The fix may lie in the data, not the representation. Tuning LayerNorm in Attention shows that tuning normalization parameters alone can bridge a modality shift and beat LoRA by over 20% on average. We therefore run both LayerNorm tuning and LoRA on exactly our stage-one and stage-two data, with matched compute. If either reaches our REVEAL camera-motion accuracy, the architectural claim fails, and we will report it as a data result. REVEAL gives a reason to expect otherwise: windowed encoders cannot compare patches across distant frames, and tuning cannot add a comparison the architecture never makes. That is a prediction, not proof.
Why not just feed optical flow from an off-the-shelf estimator? This is plausible, and we include it: flow images, encoded as an extra set of patch tokens. Our hypothesis is that raw flow patches recreate the original gap. Motion becomes another dense grid, and the language model must still separate camera from object motion. The flow-patch baseline and our no-decomposition ablation together test this. If flow patches match us, the contribution narrows to the decomposition, and we will say so.
Why not just counter language override at decoding time, for example by contrasting against a blank-video pass? REVEAL identifies two failure pathways: encoding failures and override failures, where visual evidence is present but overruled by language priors. Motion tokens address only the first. For sycophancy, a decoding fix may beat us, and we do not claim otherwise. We will report REVEAL's Video Sycophancy and Language-Only Shortcut tests separately. We will also run the late-layer probe to check whether motion signal survives deeper with our tokens. That would be evidence that structured tokens resist suppression, but it would not show they fix override.
Why not just scale the model? In REVEAL, frontier closed models still collapse on cross-frame integration. In GroundFormer, a 211M-parameter model beats a 1.2B one on grounded QA. Scale alone does not look sufficient, so we do not spend compute on this baseline beyond citing those results.
Proposed Experiments
The primary benchmark is REVEAL's five-test suite, held out from training. We generate our own synthetic camera trajectories rather than reusing REVEAL's generator outputs. The decisive check is camera-motion accuracy: we need a large rise above the 10–20% band while the LayerNorm-tuned, matched-data model stays well below us. The second decisive check is Cross-Frame Integration, where Gemini 2.5 Pro reaches 8% on masked video; we expect a clear gain over every baseline. The third is the MASH tension itself. On UNSCENE, our model with unrestricted attention should hallucinate no more than with the MASH mask. That result would show disentanglement is satisfied by separate token types, without blocking interaction inside the language model.
For transfer, we evaluate on MVBench against MASH-VLM's reported 57.6%, on NExT-GQA (accuracy and grounding), and on NExTQA against Chain-of-Glimpse's 83.3%. NExT-GQA uses GroundFormer's reported 21.5% Acc@GQA as context, though its architecture differs. We also evaluate on STSBench (dataset), which tests ego-vehicle actions and agent interactions in driving video. Ego-motion decomposition should matter most there; no reported result was found on Lacuna for STSBench, so we report all systems ourselves.
The mechanistic check repeats REVEAL's linear probes for camera-motion class, layer by layer, on the base and modified models. It measures where decodable motion information is lost.
The idea fails cleanly under any of three outcomes: the matched-data LayerNorm baseline closes the camera-motion gap, the flow-patch baseline matches the full model, or the decomposition ablation performs as well as the full model.
References
- Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
- MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations
- Learning Robust Visual-Semantic Embeddings
- Extending Multi-modal Contrastive Representations
- Shifted Chunk Transformer for Spatio-Temporal Representational Learning
- Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation
- VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
- What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA
- VideoRoPE: What Makes for Good Video Rotary Position Embedding?
- Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving