Keeping the Latent Honest: Band-Limited Skips for Sharp Stochastic Video Prediction

Predicting the next frames of a video is uncertain: a robot arm may push a block left or right, and a walking person may turn or stop. One family of models handles this uncertainty well. A recurrent variational autoencoder (a sequence model that, at every time step, draws a random "latent" code describing what might happen next and decodes it into a frame) produces different plausible futures on each run. These models are usually blurry because they are trained with a pixel-wise squared error. The obvious fixes are to add sharper loss terms (structural similarity, perceptual feature distances) and to add 3D convolutions with encoder-to-decoder skip connections, which help keep spatial detail. We argue that these two fixes, applied together, quietly defeat the purpose. Skip connections let the decoder copy detail directly from the observed past. Sharpness losses reward it for doing so. The random latent is then no longer needed, and the model turns back into a sharp but deterministic predictor.

We propose to make the decoder's shortcut to the past carry only coarse, low-frequency information, so that fine spatial detail and future motion can reach the output only through the stochastic latent. The loss is then split by pathway. Pixel and structural terms supervise the coarse, predictable content. A perceptual term is what forces the latent to carry sharp, uncertain detail. The central claim is that a recurrent variational autoencoder can be sharp and diverse at the same time if the architecture decides which information may bypass the latent. Loss weighting alone cannot make that decision.

Core Claim

In a 3D-convolutional recurrent variational autoencoder, adding structural and perceptual reconstruction losses together with full-resolution skip connections will reduce reliance on the latent. The per-step KL divergence will shrink, and samples drawn from the prior will become nearly identical. This trade-off between sharpness and diversity can be removed by band-limiting the skip pathway, meaning the skips pass only spatially downsampled, low-pass features of the context clip. Tuning the KL weight $\beta$ or reweighting the KL term cannot remove it. The claim is falsifiable. If a well-tuned $\beta$ sweep on the unrouted model reaches the same combination of sharpness and diversity, the idea fails.

Relation to Prior Work

The closest work is A Log-likelihood Regularized KL Divergence for Video Prediction with A 3D Convolutional Variational Recurrent Network. It puts 3D convolutions in every sub-network, including a 3D ConvLSTM and a truncated 3D-ResNet18. It also reweights the KL term with a maximum-likelihood term so the recurrent prior stays trained. It reports Moving MNIST MSE 39.4 at 12.9M parameters. That paper supplies our backbone and our strongest baseline. It does not use skip connections or any reconstruction term beyond pixel MSE, and it reports extra short-term stochastic noise. We add exactly the two ingredients whose interaction we want to study, and we control how they interact.

Stochastic Video Generation with a Learned Prior supplies the learned recurrent prior. Its prior's variance rises when the future becomes uncertain, and we use that behavior as a diagnostic. SVG-LP is 2D and trained with pixel loss only.

Deep End2End Voxel2Voxel Prediction shows that multi-scale 3D skip connections recover boundary detail in dense prediction. Because that model is deterministic, the bypass problem never arises there. Our proposal hinges on that difference.

On The Classification-Distortion-Perception Tradeoff proves that distortion, perceptual realism and downstream classification accuracy form a convex frontier. It also shows that pushing realism can mislead classifiers. We use it as an evaluation lens: we compare whole frontiers across methods rather than single operating points.

Lossy Image Compression with Conditional Diffusion Models is related in spirit because it separates coarse "content" latents from synthesized texture. There the split is made to trade distortion against realism in compression. Ours is made to keep a sequence model stochastic.

Conditional Temporal Variational AutoEncoder for Action Video Prediction factors prediction into structure and appearance through explicit human pose. We factor by spatial frequency, which needs no keypoints and applies to any scene.

Research Program

The model extends the Razali–Fernando architecture. A shallow hybrid 2D+3D encoder, following the half-3D blocks of CV-VAE, reads a short context clip. A 3D ConvLSTM carries state across time. A learned prior network and a posterior network produce a Gaussian latent $z_t$ at each step. The decoder receives three inputs: the recurrent state $h_t$, the latent $z_t$, and skip features from the context. The key intervention is a fixed low-pass operator $S_r$ applied to every skip tensor, a blur followed by downsampling by a factor $r$. With $S_r$ in place, the decoder cannot copy textures or fine edges from the past. The hyperparameter $r$ sets how much detail is forced through the stochastic path. Setting $r=1$ gives ordinary full-resolution skips, and $r=\infty$ removes the skips entirely.

The per-step training objective is

$$ \mathcal{L}_t = \lambda_1 \lVert B(x_t) - B(\hat{x}_t) \rVert_1 + \lambda_s \big(1 - \mathrm{SSIM}(x_t, \hat{x}_t)\big) + \lambda_p\, P(x_t, \hat{x}_t) + \beta\, \widetilde{\mathrm{KL}}\big(q(z_t \mid x_{\le t}) \,\Vert\, p(z_t \mid x_{<t})\big). $$

Here $x_t$ is the true frame and $\hat{x}_t$ the prediction. $B$ is the same low-pass operator used on the skips, so the pixel term only checks coarse content. SSIM is structural similarity. $P$ is a perceptual feature distance (LPIPS). $\widetilde{\mathrm{KL}}$ is the log-likelihood-regularized KL of Razali and Fernando, and the $\lambda$ values and $\beta$ are weights. The pixel term is band-limited on purpose. A full-resolution pixel loss would again penalize sharp but misplaced detail, which is the metric misalignment Denton and Fergus describe.

Beyond the model itself, the program produces a diagnostic we call latent reliance. For each model we decode the same context twice, once with posterior samples and once with prior samples. Latent reliance is the resulting gap in perceptual distance, together with the average pairwise perceptual distance among $K$ prior samples. A model whose latent has been bypassed shows near-zero values on both. We will trace this quantity as a function of $r$, $\lambda_p$ and $\beta$. The goal is to establish that bypass is a real, measurable failure and not a hypothetical one.

Baselines

The naive reference point is a copy-last-frame predictor together with a deterministic version of our network with $z_t$ removed, trained under the same composite loss. The deterministic model tells us how much sharpness is available with no stochasticity at all. Any stochastic model must match it on best-of-$K$ metrics and beat it on diversity to justify its existence.

The strongest existing method is the Razali–Fernando variational 3D ConvLSTM (paper), which reports Moving MNIST MSE 39.4 against 41.3 for E3D-LSTM. It is the strongest competitor because it already combines all three ingredients except skips and sharpness losses. We will first reproduce it with pixel MSE and confirm we are close to 39.4 before any comparison. We then retrain it under our composite loss with an identical tuning budget: the same grid over $\beta$ and $\lambda_p$, the same number of seeds, and matched parameter count and GPU-hours. Beating it means our model has a better frontier of FVD against sample diversity at equal or better best-of-$K$ SSIM. A better single number at one arbitrary weighting does not count.

The 2D reference is SVG-LP (paper), run with our composite loss at matched parameters. For SVG-LP on BAIR and KTH we found no reported result on Lacuna in scalar form, so we will run it ourselves.

The ablation that isolates the key ingredient is the same network with full-resolution skips, $r=1$, under the same loss and tuning grid. This is the "obvious" combination the research file points toward. If band-limiting matters, this model should show collapsed latent reliance at the loss weights where our model does not.

On Penn Action we also compare against ACT-VAE (paper), which reports FVD 1092.8 against 1318.9 for KL-VP. It is a pose-supervised upper reference, not a like-for-like competitor. Coming within its range without keypoints would be meaningful.

Why not just…?

Why not just tune $\beta$, or anneal the KL term, on the unrouted model? This is the real threat. Lowering $\beta$ does raise the KL, but a high KL does not guarantee that the decoder uses the latent when full-resolution skips offer a cheaper route to low perceptual loss. The posterior can encode information that the decoder ignores. The $\beta$ sweep in our ablation settles this directly. If some $\beta$ for the $r=1$ model reaches our frontier, band-limiting is unnecessary and we will report that.

Why not just drop the skips and keep the Razali–Fernando bottleneck with sharper losses? Removing skips ($r=\infty$) also removes the bypass. However, the V2V results (paper) suggest the recurrent bottleneck then has to carry static appearance as well, which is the memorization burden the file notes for high-dimensional scenes. Our sweep over $r$ includes $r=\infty$. The prediction is that intermediate $r$ is better than both ends, and if it is not, the simpler model wins.

Why not just use a deterministic 3D predictor with perceptual loss? On KTH, much of the motion is nearly predictable, and a sharp deterministic model may score well on single-sample SSIM. Its weakness shows on BAIR and Stochastic Moving MNIST, where the future branches. Both Denton–Fergus and Razali–Fernando report that deterministic baselines blur or lose objects that leave the frame. We report per-dataset results precisely so that a KTH win for the deterministic model is visible rather than averaged away.

Why not just evaluate best-of-$K$ with plain MSE training? Best-of-$K$ scoring rewards diversity, but it does nothing to make individual samples sharp. Our frontier plots report mean-of-$K$ and best-of-$K$ side by side, so a model that is diverse only because it is noisy will show a poor mean-of-$K$ perceptual score.

Proposed Experiments and Decisive Checks

We train on Stochastic Moving MNIST, KTH Actions and BAIR Robot Pushing. Each model sees 10 context frames and predicts 10 frames during training. At test time we roll out to 30 frames on BAIR and up to 40–60 frames on KTH, matching the horizon Razali and Fernando report. Metrics are MSE, PSNR and SSIM (reported as best-of-$K$ and mean-of-$K$ with $K=100$), FVD, pairwise perceptual diversity, and per-step KL together with latent reliance.

Following the classification–distortion–perception lens, we add a utility axis on KTH. A frozen C3D-style action classifier (paper) is run on the predicted clips. This checks whether perceptual sharpening introduces artifacts that mislead recognition, which is the failure the tradeoff theory predicts.

The first decisive check is the collapse test. At the loss weights that give the best FVD, the $r=1$ model should show latent reliance near zero and a fall in perceptual diversity relative to our model. If this fails to appear across the $\beta$ grid, the motivating phenomenon does not exist and the proposal stops there.

The second decisive check is the frontier comparison. Across the matched hyperparameter grids, our model's frontier of FVD against diversity should dominate those of the Razali–Fernando baseline and the $r=1$ ablation on BAIR and Stochastic Moving MNIST. This must hold without losing best-of-$K$ SSIM on KTH.

The third check tests whether the learned prior still behaves as a prior. In SVG-LP, the prior's variance spikes at bounces and contact events, and it should keep doing so under the composite loss. A flat variance trace would mean the latent survives only as injected noise, which is a weaker result than the claim.