Seer: Language Instructed Video Prediction with Latent Diffusion Models

Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, Yang Gao

TL;DR

Seer is a highly efficient text-conditioned video prediction framework that generates high-fidelity, temporally coherent future frames by "inflating" pretrained text-to-image (T2I) stable diffusion models along the temporal axis. By introducing a novel text decomposer that breaks global instructions into frame-specific sub-instructions and utilizing localized spatial-temporal attention, Seer achieves state-of-the-art performance while requiring a fraction of the training compute of traditional video generation models.

What Problem Does This Paper Address?

Text-conditioned Video Prediction (TVP) aims to predict future video frames given an initial reference frame and a natural language instruction. This capability is vital for robotics—allowing an agent to visualize the outcome of a plan—and video editing.

However, current TVP approaches face three major roadblocks:

  1. Extreme Computational and Data Demands: Training video diffusion models from scratch requires massive datasets of paired text-video clips and thousands of GPU hours.
  2. Low Fidelity: Generated frames are frequently blurry, losing the fine details of the background or objects present in the initial frame.
  3. Coarse Language Guidance: Global text prompts (e.g., "tipping a beer can") describe an entire action sequence but fail to provide the frame-by-frame granularity needed to guide a generative model through the sequential phases of that action.
Seer uses natural language instructions and reference frames to predict future frames.

Methods

Seer tackles these challenges by adapting a pretrained 2D Latent Diffusion Model (LDM) into a 3D video generation network, focusing training strictly on lightweight temporal layers while keeping the spatial layers frozen.

The architecture consists of two primary innovations:

1. Data & Computation-Efficient 3D Network

Instead of training a video model from scratch, the authors "inflate" a pretrained 2D U-Net. They expand 2D convolution kernels into 3D counterparts (e.g., $3 \times 3$ becomes $1 \times 3 \times 3$) and introduce a Scaled Autoregressive Window Temporal Attention (SAWT-Attn) mechanism. SAWT-Attn restricts self-attention to local spatial windows across the temporal sequence using a causal mask, ensuring that pixels propagate coherently from past frames to future frames without exploding computational complexity.

2. Frame Sequential Text (FSText) Decomposer

To fix the issue of coarse language guidance, the authors introduce the FSText Decomposer. This module takes a global text embedding from a frozen CLIP encoder and automatically dissects it into a sequence of fine-grained sub-instructions—one for each frame.

Crucially, the authors use an identity initialization strategy for this decomposer. By pre-training the module to act as an identity function before the diffusion process begins, they ensure that the generated sub-instructions stay within the well-behaved embedding space of the pretrained T2I model rather than drifting into chaotic, unaligned representations. These frame-specific text tokens are then injected into the U-Net via frame-individual cross-attention (fic-attn).

Figure 2: (a) Seer’s pipeline includes an Inflated 3D U-Net for diffusion and a Frame Sequential Text Transformer for text conditioning. (b) Our Inflated 3D U-Net expands the pre-trained 2D Conv kernel to 3D kernels and connects with the Scaled Autoregressive Window Temporal Attention layer.

Datasets

The authors evaluated Seer across several challenging action and robotics datasets:

  • Something-Something V2 (SSv2): A large-scale dataset of humans performing basic interactions with everyday objects.
  • BridgeData: A dataset of robotic manipulation tasks in kitchen environments.
  • EpicKitchens-100: An egocentric dataset tracking daily kitchen activities.
  • EGO4D: Used exclusively for zero-shot generalizability testing.
  • UCF-101: Used for standard class-conditioned video prediction baselines.

Key Insights and Ideas

  • Pretrained T2I Models are Video Goldmines: You don't need to learn spatial representations from scratch for video. Freezing the 2D spatial layers of a stable diffusion model preserves rich visual priors, meaning the network only needs to learn how things move, drastically cutting down training time.
  • Temporal Text Decomposition Prevents Semantic Override: When using global text prompts, a strong text prior can sometimes "override" the visual reality of the reference frame, causing the model to hallucinate entirely new backgrounds. Breaking the prompt down into implicit temporal sub-steps forces the model to align its generation with the current state of the video.
  • Smart Initialization is Mandatory: Randomly initializing modules that interface with pretrained latent spaces often destroys the prior alignment. Forcing the text decomposer to start as an identity function acts as a regularizer that preserves the fragile text-to-image grounding.

Key Results

  • Compute Efficiency: Seer required only 480 GPU hours (120 hours on 4 $\times$ RTX 3090 GPUs) to train, compared to SOTA models like CogVideo which demand over 12,480 GPU hours.
  • State-of-the-Art Metrics: On the SSv2 dataset, Seer achieved a Fréchet Video Distance (FVD) of 112.9, a massive 31% improvement over the previous state-of-the-art model, Tune-A-Video (FVD of 291.4).
  • Human Preference: In blind A/B testing for video manipulation quality, human evaluators preferred Seer's generations 72.2% of the time over Tune-A-Video, and roughly 90% of the time over TATS and MCVD.

Limitations and Future Work

Despite its strengths, Seer exhibits a few notable limitations:

  • Long-Horizon Appearance Drift: When generating longer video sequences, the model can struggle with appearance consistency, occasionally altering object colors or textures (e.g., changing the cover design of a falling book mid-video).
  • Egocentric Bias: On egocentric datasets like EpicKitchens-100, Seer occasionally prioritizes predicting camera ego-motion (novel view synthesis) over actual human-object interaction.
  • Long Video Extension: Simple interpolation or repetition of sub-instructions to extend videos beyond the trained frame length introduces unexpected noise or static motion, indicating a need for more robust autoregressive text-planning strategies in future iterations.

Summary updated

Abstract

Imagining the future trajectory is the key for robots to make sound planning and successfully reach their goals. Therefore, text-conditioned video prediction (TVP) is an essential task to facilitate general robot policy learning. To tackle this task and empower robots with the ability to foresee the future, we propose a sample and computation-efficient model, named \textbf{Seer}, by inflating the pretrained text-to-image (T2I) stable diffusion models along the temporal axis. We enhance the U-Net and language conditioning model by incorporating computation-efficient spatial-temporal attention. Furthermore, we introduce a novel Frame Sequential Text Decomposer module that dissects a sentence's global instruction into temporally aligned sub-instructions, ensuring precise integration into each frame of generation. Our framework allows us to effectively leverage the extensive prior knowledge embedded in pretrained T2I models across the frames. With the adaptable-designed architecture, Seer makes it possible to generate high-fidelity, coherent, and instruction-aligned video frames by fine-tuning a few layers on a small amount of data. The experimental results on Something Something V2 (SSv2), Bridgedata and EpicKitchens-100 datasets demonstrate our superior video prediction performance with around 480-GPU hours versus CogVideo with over 12,480-GPU hours: achieving the 31% FVD improvement compared to the current SOTA model on SSv2 and 83.7% average preference in the human evaluation.
BibTeX
@inproceedings{gu2024seer,
  author = {Xianfan Gu and Chuan Wen and Weirui Ye and Jiaming Song and Yang Gao},
  title = {{Seer: Language Instructed Video Prediction with Latent Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=qHGgNyQk31},
}
Figure