Patch Tokens That Follow the Content: Trajectory-Aligned Temporal Fibers for Video Models

Video models built from image encoders often know what is in a clip but not how it moves. They score well on action labels that one frame can reveal, and fall to chance when asked which way something moved, how fast, or in what order events happened. One recent fix lines up each image patch with the patch at the same pixel position in the next frames and runs a small temporal model along that "fiber." That works when the camera is still and objects move slowly. When the camera pans or an object crosses the frame, the fiber at a fixed position sees a stream of unrelated content, and the motion signal is lost again.

We propose that the patch grid should move with the content. Each frame is already cut into patches by a vision transformer, an image model that splits a picture into small square tiles and processes them as a sequence of tokens. We use those tiles as units to track. For each patch, a cheap nearest-neighbour match in the encoder's own feature space finds where its content went in the next frame. This produces short patch trajectories at no extra detector cost. The temporal module then reads two things along each trajectory: the features, and the trajectory's displacements. Speed and direction stop being something the language model must infer from appearance and become explicit geometry in the token stream. The token budget is the same as fixed-position fibers.

Core Claim

Fixed-position temporal fibers fail mainly because they assume patch content stays in place, not because the temporal module lacks capacity. Replacing them with fibers that follow matched patches, and adding the matched displacements as input, should close a large part of the direction, speed, and event-order gap on TempCompass and the agent-to-agent gap on STSBench. It should do so at equal token count and without object detectors. The claim is specifically about following the content. Where the correspondences come from is a design variable that we test, not the contribution itself.

Where This Sits

Three papers supply the evidence for the gap. TempCompass builds reversed, concatenated, and reordered videos and shows video language models collapse to 25–33% on speed and direction, often below a single-frame image model. STSBench shows that text-only GPT-4o given ground-truth 3D trajectories (57.08%) beats every pixel-based driving model, including InternVL 2.5 8B (46.07%). That result is the strongest hint that explicit trajectories are what the visual side fails to deliver. Something-Else shows that reasoning over tracked box geometry resists appearance bias, but it relies on Faster R-CNN detections and fails on non-rigid actions like tearing.

The method we build on and depart from is TempFlex. Its Temporal Fiber Fusion runs local, memory, and periodic paths over features at a fixed spatial location across frames. We keep that module but change what a fiber is. A fixed-position fiber records how the scene changes at one point in the image. Our fiber records how one piece of content changes as it moves.

Two other papers constrain the design. Boosting Video Super Resolution with Patch-Based Temporal Redundancy Optimization finds that per-patch temporal processing of static regions injects flow errors. We therefore let near-zero-displacement trajectories collapse to a single token rather than propagating noise. K-centered Patch Sampling shows that a spatiotemporal bag of patches beats frame sampling, but it discards grid structure and uses non-learned selection. Our trajectories keep each patch's identity across time, which a bag cannot.

ViDiDi trains encoders on whole-frame temporal differences. A difference taken at a fixed position mixes motion and appearance change together. A difference taken along a trajectory separates the two: the displacement carries the motion, and the feature difference carries the change in appearance.

Research Program

The encoder is a frozen-then-lightly-tuned SigLIP-style patch encoder inside an open MLLM (InternVL 2.5 8B, chosen because STSBench reports it). For frames $t$ and $t+1$, each patch $i$ at grid location $p_i^t$ is matched to the patch in a local $k\times k$ window of frame $t+1$ that maximises cosine similarity of encoder features. The match uses a soft-argmax so it stays differentiable, and a forward–backward consistency check marks occluded matches as track ends. Chaining matches gives trajectories of length up to $T$.

To remove camera motion, the median displacement of the frame is subtracted, giving relative motion. This matters for driving, where the whole image moves. Each trajectory yields a feature sequence and a displacement sequence $\Delta p^t = p^{t+1}-p^t$. The displacements are embedded by a small MLP and added to the TFF input. Static trajectories, those with $\|\Delta p\|$ below a learned gate, are pooled to one token. The output keeps TempFlex's per-fiber token count.

Training uses the same instruction data as the fixed-fiber baseline. Matching costs $O(Nk^2)$ dot products per frame pair for $N$ patches, which is small next to encoder cost. We will report this cost exactly.

The analysis has three parts. We stratify TempCompass and SSv2 accuracy by measured camera-motion magnitude, where the claim predicts the gain grows with motion. We probe whether trajectory tokens linearly decode object velocity. And we inspect the failures on non-rigid actions from the Something-Else taxonomy, where patch matching may degrade like box tracking does.

Baselines

The naive reference points are a single middle frame fed to the same MLLM, and mean-pooled per-frame tokens. TempCompass already shows these are close to how current video LLMs behave. Any method that fails to beat the single-frame model on direction and speed has not addressed temporal loss.

The strongest direct competitor is TempFlex's fixed-position Temporal Fiber Fusion, because it is the only method in this literature that adds per-patch temporal modelling to an MLLM at a fixed token budget. We reimplement it inside the same backbone, with identical training data, steps, frame count, and hyperparameter search budget, so that the only difference is fiber definition plus displacement input. No reported result was found on Lacuna for TempFlex's per-dimension TempCompass scores; the paper reports only "state-of-the-art or competitive." Our reimplementation therefore defines the bar. Beating it means a gain on the direction and speed dimensions clearly outside seed variance (three seeds), with no loss on the action dimension.

On recognition benchmarks the comparators are K-centered Patch Sampling (Motionformer top-1 on Kinetics-400 from 71.42% to 74.85%, and a 2.86% relative gain on SSv2) and Something-Else (STIN+I3D ensemble at 58.1% top-1 on the compositional split). On SSv2 and Something-Else we attach our trajectory module to a plain video transformer of matched size and match its token budget to K-centered sampling. Beating STIN+I3D without a detector or an ensemble would be a strong result; matching it would still be informative.

On STSBench, InternVL 2.5 8B at 46.07% is the pixel baseline. Text GPT-4o with ground-truth trajectories at 57.08% is a ceiling reference, not a competitor.

The key ablations isolate the main ingredient. First, the full model with matching replaced by identity, which is exactly fixed fibers. Second, matched fibers without displacement tokens. Third, displacement tokens with trajectories shuffled across patches. If shuffling hurts little, the gain is not coming from correspondence.

Why not just…?

Why not just warp features with off-the-shelf optical flow before the fiber module? This is the real threat. It may work as well, and if it does, that supports the core claim, because content-following is what matters. It would also make the encoder-feature matching an engineering choice rather than a necessity. We run flow-warped fibers (dense flow resampled to the patch grid) as a full variant under the same budget. Boosting Video Super Resolution shows flow errors compound in static and irregular regions, which suggests feature matching may be more robust on DTVIT-style irregular motion. We will report accuracy and wall-clock cost for both and recommend whichever wins.

Why not just use full joint space-time attention, which can already find correspondences? In principle it can. In practice the video LLMs that TempCompass tests attend over all frame tokens and still hit chance on direction, and joint attention costs quadratically in tokens. We include a joint-attention baseline at matched token count. Our prediction is that explicit displacement inputs beat implicit attention-based correspondence on speed and direction. If they do not, the proposal's value reduces to efficiency.

Why not just track detected objects, as Something-Else does? Detector tracks give clean geometry but miss untracked classes and non-rigid changes. Something-Else needed an I3D ensemble to reach 58.1%, and STSBench driving models still lose 20–30% on agent-to-agent interactions. Patch trajectories cover every region. The per-category Something-Else breakdown will show whether this helps on tearing and poking classes or fails there too.

Why not just feed temporal differences, as ViDiDi does? Fixed-position differences mix up camera motion, object motion, and appearance change. We test frame-difference and second-difference inputs to the fixed-fiber model as a variant. The motion-stratified analysis should show those inputs degrading under camera pans, where relative trajectories should not.

Decisive Checks

The idea is supported if three things hold. Matched fibers beat fixed fibers on TempCompass direction, speed, and event order, with the gap growing with camera motion. The shuffled-trajectory ablation loses most of that gain. And STSBench agent-to-agent accuracy rises measurably above the 46.07% baseline toward the trajectory-oracle reference.

The idea is refuted if fixed fibers with displacement-free matching, or joint attention at equal tokens, do as well. In that case content-following is not the bottleneck, and the failures documented by TempCompass lie elsewhere, for example in the language model's use of visual tokens.