Trackable Latents for Video Generation

Most video generators work in two stages. First an autoencoder compresses clips into a small grid of numbers called latents. Then a separate generative model, usually a diffusion model (a generator that learns to turn random noise into samples by denoising it step by step), learns to produce new latent grids. The autoencoder is almost always trained only to reconstruct pixels. Nothing in that training asks a latent cell describing a dog's ear at one moment to be findable among the latent cells of the next moment. This proposal adds exactly that requirement. The autoencoder is trained so that content can be tracked forward through its latents and then back to where it started. The claim is that latents shaped this way give the generator smoother, more stable motion, and that they do so without the usual cost of making videos more static.

The reason to expect this comes from how temporal coherence has been enforced so far. The simplest approach, penalizing differences between consecutive latents, rewards stillness. A model can satisfy it by not moving, and benchmarks such as VBench document this pattern: temporal consistency scores rise as dynamic degree falls. A forward-backward tracking loop is indifferent to how far content moves. It asks only that content stay identifiable wherever it goes. If that difference holds once the loss is moved from discriminative trackers into the latent space of a generator, it gives a direct handle on the consistency-versus-motion trade-off.

Core Claim

A cycle-consistent tracking loss applied to the Stage 1 video autoencoder, with the diffusion loss blocked from flowing back into the encoder, will do three things relative to a reconstruction-only autoencoder of the same size:

  1. It will improve subject consistency and flicker metrics in generated video.
  2. It will leave dynamic degree unchanged or higher.
  3. It will speed up convergence of the generative model.

A slowness penalty of matched strength, by contrast, should buy consistency only by lowering dynamic degree. The specific new element is the contrast between "trackable" and "slow" as properties of a generative latent space, tested at matched compute.

Relation to Prior Work

Three papers motivate this by showing that latent geometry, not only reconstruction quality, governs how well a generative prior trains.

REPA-E aligns diffusion-transformer hidden states to frozen DINOv2 features. It also shows that letting the diffusion loss flow into the autoencoder oversimplifies the latents, and it fixes this with a stop-gradient. Its reported results are 17x to 45x faster convergence and an ImageNet 256 FID of 1.26. REPA-E is purely image-based, though, and its alignment target is static.

V-RAE builds video latents on frozen foundation models such as V-JEPA and DINOv3. It reports gFVD 19.16 on Kinetics-600, against 52.25 for Wan2.2 and 51.83 for CogVideoX. However, it imports temporal structure from the frozen teacher rather than training for it, and it puts heavy demands on a large transformer decoder.

CV-VAE inflates an image VAE to video with 4x temporal compression and trains with reconstruction and decoder alignment only. Its downstream frame-consistency gain on VideoCrafter2 is small (0.292 to 0.301), which suggests that architecture alone does not fix flicker.

This proposal supplies what all three omit: an explicit, teacher-free temporal objective inside a compact reconstructive latent. It also keeps REPA-E's lesson that the diffusion gradient must not reshape the encoder.

The method comes from the tracking literature. Temporal cycle consistency tracks a patch forward and then backward and penalizes failure to return to the start. SSTrack shows that forward-backward and contrastive objectives avoid the collapse that plain distance minimization causes, reaching 72.4% Average Overlap on GOT-10k. It is discriminative, however, and has no decoder or prior.

Two papers constrain the design. Densemarks gets strong dense correspondence from CoTracker3 point tracks, with MAE 3.68 against 7.60 for DINOv3. This proves that tracks are a rich signal but also shows the dependence on a teacher. GeoVideo adds geometric reprojection losses to a video diffusion transformer and raises Multi-View Consistency Score (MVCS) on DL3DV from 61.2 to 77.2. It needs depth and pose pseudo-labels and masks out exactly the non-rigid foreground motion where flicker is worst. Our loss needs no teacher and applies equally to cloth, hair and limbs.

Method

The encoder produces latent frames $z_1,\dots,z_T$. Each $z_t$ is a set of $N$ spatial feature vectors, which we L2-normalize. Between adjacent latent frames we form a soft transition matrix $$A_{t\to t+1} = \mathrm{softmax}\!\left(z_t z_{t+1}^{\top}/\tau\right),$$ where the softmax runs over the $N$ positions of frame $t+1$ and $\tau$ is a temperature.

Chaining these matrices forward to frame $t+k$ and then backward gives a round-trip matrix $W$. The cycle loss is $$\mathcal{L}_{\text{cyc}} = -\frac{1}{N}\sum_{i}\log W_{ii}.$$ This is the probability mass that each position's walk returns to itself, which is the contrastive random-walk form of cycle consistency.

The Stage 1 objective is the usual reconstruction, perceptual and KL terms plus $\lambda\,\mathcal{L}_{\text{cyc}}$.

One loophole must be closed. An encoder that writes absolute position into every cell would close every cycle trivially. To prevent this, each frame in the walk is drawn as an independently jittered crop, so the network can only return home by matching content.

Stage 2 is a standard latent diffusion transformer, trained on frozen Stage 1 latents. In a secondary arm it is jointly tuned under a REPA-E-style stop-gradient.

Research Program

The program has three parts.

  1. Build and measure trackability. We build the autoencoder and measure trackability as a standalone property of a latent space. To do this we use the latents directly as features for DAVIS-2017 label propagation, which gives a $\mathcal{J}\&\mathcal{F}$ number per latent space. As references, self-supervised VOS reports 70.7 on this task with dedicated machinery, and MAST reports 65.5. We do not expect a generative latent to reach those numbers. We do expect it to rank latent spaces.
  2. Test whether trackability predicts generation quality. We train generative priors on each latent space under identical budgets.
  3. Test the trade-off claim directly. We sweep the weight of each temporal loss and trace consistency against dynamic degree.

Baselines

The do-nothing reference is the same autoencoder architecture trained with reconstruction losses only. This is a CV-VAE-style 3D autoencoder with 4x temporal compression and 16 channels, followed by an identical diffusion transformer. Beating it means better gFVD and VBench Subject Consistency with Dynamic Degree within noise or higher.

The strongest existing method is V-RAE's frozen-foundation latent. It is the strongest competitor because it already contains temporal structure from V-JEPA and has the best reported Kinetics-600 generation figure in the file, gFVD 19.16. We rerun it at our resolution and compute, using its published recipe, with the same diffusion transformer and the same number of training steps. We report its decoder parameters and FLOPs next to ours, because a heavier decoder is part of its cost. Beating it means matching its gFVD within a small margin with a substantially lighter decoder and better reconstruction (rFVD), or exceeding it outright.

Two further baselines isolate the key ingredient. The first is a slowness penalty, squared distance between consecutive latents, at weights swept to match the cycle loss's effect on Subject Consistency. The second is a teacher-supervised variant in which CoTracker3 tracks, as in Densemarks, define positive pairs in place of the self-supervised walk. All arms share data, steps, learning-rate search and seeds. Three seeds are run for the generation metrics.

Why not just…?

Why not just use frozen foundation-model latents, as V-RAE does? This is the main threat, and we treat it as one. V-RAE already gives fast convergence and coherent motion. Our experiments will measure DAVIS trackability of V-RAE latents alongside ours. If the two are equally trackable and generate equally well, then the value of the cycle loss reduces to making compact reconstructive latents competitive with a lighter decoder. We will report that outcome as it is rather than claim more.

Why not just penalize latent change between frames? The file notes that this objective collapses toward constant latents, and VBench++ shows that consistency gained this way tends to come from reduced dynamic degree. Our sweep tests this directly. If slowness at matched consistency shows equal dynamic degree, the central contrast of this proposal is false.

Why not just do video REPA, aligning the diffusion transformer to V-JEPA features? This is a plausible and cheap alternative that REPA-E's success invites. It acts on the generator rather than the latent space, so it is complementary, not exclusive. We run it as an arm both alone and combined with trackable latents, and report whether the gains add.

Why not just supervise with an off-the-shelf point tracker? Tracks from a teacher inherit its drift and occlusion errors, and the file identifies this as the weakness of Densemarks- and GeoVideo-style supervision. The teacher-supervised baseline settles it empirically. If it wins, the self-supervised walk is unnecessary.

Proposed Experiments

The data are class-conditional Kinetics-600 and UCF101 at 128px with 17-frame clips. A text-conditional run on VideoUFO checks transfer.

The metrics are:

  • generation quality: gFVD;
  • reconstruction: rFVD and PSNR;
  • latent smoothness: V-RAE's tFVD;
  • temporal behavior: VBench Subject Consistency, Temporal Flickering, Motion Smoothness and Dynamic Degree;
  • latent trackability: DAVIS-2017 $\mathcal{J}\&\mathcal{F}$ of latent features;
  • semantics: UCF101 linear probe. V-RAE reports 89.13% for its latents against 30.83% for a traditional VAE.

There are three decisive checks.

  1. Frontier shift. On the Subject Consistency versus Dynamic Degree plane, the cycle-loss sweep must sit above the slowness sweep, not slide along the same curve.
  2. Trackability predicts generation. Across all trained latent spaces, latent DAVIS score should rank-correlate with gFVD and flicker. A null correlation would undercut the mechanism even if the method happens to help.
  3. No reconstruction tax. rFVD of the trackable autoencoder must stay within a small margin of the reconstruction-only baseline. If tracking costs fidelity, the gain may simply be the over-smoothing that REPA-E warns about, arriving by a different route.