Most of a video is a copy of the previous frame. Yet a flow-matching video generator (a model that learns a velocity field carrying random noise into data along a continuous path) usually starts every new frame from pure noise. It then spends equal effort on a static wall and on a moving hand. Two papers already start the flow from the previous frame plus a little noise instead of from scratch: Efficient Video Prediction via Sparsely Conditioned Flow Matching (RIVER) for video and WarmPrior for robot actions. Both report shorter, straighter paths and far fewer integration steps. In both, however, the noise level is a single scalar applied uniformly to every pixel or action dimension.
We propose to make that noise level a per-token map, chosen by the model, so that each frame's transport costs as little "kinetic energy" (the integrated squared velocity, a standard measure of how much work the flow does) as the uncertainty of the future allows. Tokens the model predicts will not change get essentially zero noise and zero velocity. They can be copied forward and dropped from the expensive computation entirely. Tokens with uncertain futures get enough noise to avoid trajectory crossings. To make this trainable, the work is split into two stages. Spatial layers first learn, on still images, to clean up spatially varying noise. Temporal layers then learn, on video, where the uncertainty lies. In this design, kinetic energy is the objective that sets the noise map. Temporal redundancy becomes the compute saving that follows from it. Decoupled training is what lets the image-trained layers handle arbitrary noise maps.
Core Claim
Consider generating frame $k+1$ given frame $k$, with source $x_0 = x_k + \sigma \odot \epsilon$. Here $\sigma$ is a per-token noise-scale map, $\epsilon$ is standard Gaussian noise, and $\odot$ is elementwise product. Along the straight conditional path to the target $x_{k+1}$, the expected kinetic energy is
$$ E(\sigma) = \sum_i \left( \mathbb{E}\|x_{k+1,i} - x_{k,i}\|^2 + d\,\sigma_i^2 \right), $$
where $i$ indexes tokens and $d$ is the token dimension. The cross term vanishes because $\epsilon$ is independent of the frames.
Kinetic energy alone pushes every $\sigma_i$ to zero. When the future of token $i$ is multimodal, though, a near-zero $\sigma_i$ makes conditional paths from nearly identical sources fan out to different targets. The aggregate flow then bends, and few-step samplers fail. This is the irreducible "Reynolds-stress" curvature identified by Isokinetic Flow Matching, and the "branching cost" analyzed by WarmPrior. The right $\sigma_i$ therefore balances transport length against path curvature, token by token.
We train a small temporal head to predict $\sigma$ by minimizing
$$ \mathcal{L}_\sigma = E(\sigma) + \lambda \sum_i \mathbb{E}_t \left\| \frac{D v_i}{Dt} \right\|^2 . $$
The material derivative $Dv/Dt = \partial_t v + (v\cdot\nabla)v$ is particle acceleration. We estimate it with Iso-FM's stop-gradient finite-difference lookahead, which that paper reports adds under 25% training overhead. The weight $\lambda$ sets the trade-off.
The claim is that this learned map yields better quality per unit of compute than any uniform noise level. There are two separate reasons. Well-placed noise lets fewer integration steps suffice. Tokens with $\sigma_i \approx 0$ and near-zero predicted velocity can be skipped at every step.
Research Program
Stage one (spatial, images only). We train the spatial transformer layers on single frames and image data to transport $x + \sigma\odot\epsilon \to x$ for random, spatially structured $\sigma$ maps. These include blobs, masks, and smooth fields, plus uniform maps as a special case. This follows the spatial-first curricula of ChronoDepth and BEVT, which show that image-grounded spatial layers cut video training cost and reduce flicker. Our twist is the training task. The spatial layers learn a "local restoration under heterogeneous uncertainty" skill that the temporal stage will call on. They do not learn generic denoising.
Stage two (temporal, video). We freeze the spatial layers. We then train temporal attention layers and the $\sigma$-predicting head on clips, with the previous frame as source. To match RIVER, we keep its conditioning on one randomly sampled earlier context frame. This lets us attribute differences to the source design rather than to the conditioning.
Freezing matters for a specific reason. Iso-FM's penalty, applied to a jointly trained model, can trade appearance quality for straightness. MoTrans and Mode Seeking meets Mean Seeking document such motion-appearance gradient interference. With spatial layers frozen, the curvature penalty can only reshape motion and noise placement.
Inference. At inference, any token with $\sigma_i < \tau$ and velocity norm below $\tau$ at the first integration step is copied from frame $k$. It is removed as a query from all later steps. It remains available as a cached key, so information is not lost irreversibly, which avoids the failure DyCoke attributes to one-shot pruning.
Analysis. We will test whether the learned $\sigma$ map tracks true future change. We measure its rank correlation with per-token $\|x_{k+1}-x_k\|$ on held-out clips. We also measure its correlation with the spread of multiple sampled futures, which tells us whether the map captures uncertainty and not merely motion magnitude. Finally, we will check whether per-token kinetic energy stays bounded near $t=1$. A Kinetic Energy Perspective of Flow Matching links terminal energy blow-up to memorization, and a $\sigma$ map that collapses too far could trigger exactly that.
Baselines
Naive reference. The same backbone trained with standard flow matching from a pure Gaussian source, sampled with matched step counts. It shows how much any warm start buys.
Strongest existing method: RIVER. It uses a noised previous frame with a single global noise level and sparse context conditioning. On BAIR Robot Pushing, RIVER reports an FVD of 106 using 625 GB×Hours of compute and up to a 10× reduction in sampling steps. We will reimplement RIVER on our backbone, with the same data, the same total training GPU-hours, and the same conditioning. Its scalar noise level gets a sweep budget equal to the tuning budget we spend on $\lambda$ and $\tau$. We will also report its published configuration.
Long-horizon reference: CSSC. For long rollouts, the reference is Causal State Compression, which reports 64-frame BAIR FVD of 96.4 against 218.7 for its autoregressive baseline. CSSC addresses drift through memory rather than through the source distribution. It is complementary, so we also test our method with its keyframe memory added.
Beating RIVER. Our method beats RIVER if it reaches equal or lower FVD at equal or fewer total transformer FLOPs per generated frame. Alternatively, it beats RIVER if it shows clearly lower FVD at matched FLOPs across the whole quality-compute curve, not just at a single operating point.
Ablations isolating the key ingredient.
- Uniform noise plus Iso-FM penalty. This tests whether straightening alone explains the gains.
- Per-token noise from a heuristic motion map. This uses the absolute difference between the last two observed frames, with no kinetic objective.
- Learned motion-mask head. This head is trained with a binary "did this token change" loss instead of $\mathcal{L}_\sigma$.
- Joint training. The full method with spatial layers unfrozen, to test whether decoupling matters.
- Stage one with uniform noise only. This tests whether heterogeneous-noise image pretraining is needed.
Why not just…?
Why not just tune RIVER's single noise level and add Iso-FM's penalty? This is the most serious threat. It gets warm starts and straight paths with no new machinery. A single scalar must, however, be large enough for the most uncertain token in the frame. Under the energy decomposition above, that wastes transport on every static token and forbids skipping them. Whether the waste matters in practice is empirical. The "uniform plus Iso-FM" ablation is designed to settle it. If that ablation matches our quality-versus-FLOPs curve on BAIR and KTH, the per-token map is not worth having and we will say so.
Why not just skip tokens whose pixels did not change between the last two frames, as DyCoke-style merging does? Past change is a poor proxy for future uncertainty. A robot gripper at rest just before a grasp shows no recent motion, yet that is the region about to change most. KeyWorld is built around exactly such kinematic turning points. The heuristic-map ablation measures this directly. We will report FVD restricted to frames around contact events, where we expect the heuristic to fail. If it does not fail there, the learned map adds little.
Why not just distill the model into a four-step sampler? Distillation, as in Mode Seeking meets Mean Seeking's 4-step generator, reduces the number of steps. It still runs every token at every step, and it adds a teacher and a second training phase. Our approach attacks both steps and tokens. The two are compatible, so the fair test is whether distillation alone, or distillation plus our source design, gives the better curve at matched FLOPs.
Why not just train everything jointly? Joint training is simpler. The cited decoupling results, however, concern appearance damage from motion gradients, and a curvature penalty is precisely a motion-side gradient. The joint-training ablation will show whether that damage appears. We will measure per-frame FID on generated frames alongside FVD.
Proposed Experiments
Datasets. The main testbed is BAIR Robot Pushing at 64×64, which has a published RIVER FVD of 106 and a 64-frame CSSC FVD of 96.4. KTH and CLEVRER, both used by RIVER, add human motion and collision events. The file has no reported RIVER number on Lacuna for either dataset, so we will rerun RIVER on both. For larger, more varied motion, we use Something-Something V2; no reported result found on Lacuna for a 64-frame rollout number there either.
Metrics. We report:
- FVD at 16 and 64 frames;
- per-frame FID and PSNR over time;
- the number of function evaluations;
- the fraction of tokens computed per step;
- measured wall-clock time and FLOPs per frame.
Decisive checks. There are three, in order of importance:
- Our method must dominate tuned uniform-noise-plus-Iso-FM on the FVD-versus-FLOPs curve on BAIR.
- The learned $\sigma$ map must correlate with sampled-future spread more strongly than the heuristic motion map, especially near contact events.
- Over 64-frame rollouts, error from copied tokens must not accumulate. To check this, we compare drift curves with and without CSSC memory against CSSC's reported 96.4.
If the first check fails, we abandon the method. If only the third fails, the method still works for short-horizon prediction, but long rollouts need a periodic full-recompute refresh.
References
- Efficient Video Prediction via Sparsely Conditioned Flow Matching
- WarmPrior: Straightening Flow-Matching Policies with Temporal Priors
- Isokinetic Flow Matching for Pathwise Straightening of Generative Flows
- Learning Temporally Consistent Video Depth from Video Diffusion Priors
- BEVT: BERT Pretraining of Video Transformers
- MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
- Mode Seeking meets Mean Seeking for Fast Long Video Generation
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
- A Kinetic Energy Perspective of Flow Matching
- Causal State Compression for Long-Horizon Video World Models: A Bounded-Drift Theory and Efficient Architecture
- KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models