TL;DR
PixVid is a simple yet powerful recipe for "lifting" pretrained latent video diffusion models directly into pixel space. By replacing the restrictive VAE with a frame-wise convolutional head, PixVid enables $2\times$ higher resolution at no extra token cost, slashes interactive latency from 495ms to 34ms, and eliminates the motion blur common in temporally compressed latent models.
What Problem Does This Paper Address?
Most state-of-the-art video generation models (like Sora or Wan2.1) don't work with pixels directly. Instead, they use a Variational Autoencoder (VAE) to compress video into a "latent space." While this makes training computationally feasible, it introduces three major bottlenecks:
- The Resolution Tax: Doubling the resolution quadruples the number of latent tokens, making high-definition video exponentially expensive to generate.
- Temporal Lag: Many VAEs compress time, meaning several frames are squashed into a single latent. In interactive applications (like games), the model can't respond to a user's input until an entire "block" of frames is ready, leading to noticeable lag.
- Motion Artifacts: Because multiple frames share one latent representation, fast-moving objects often appear blurred or "morphed" as the model struggles to represent distinct positions within a single compressed token.
The authors of PixVid argue that we should go "back to pixels" to overcome these structural limitations.
Methods
The core idea of PixVid is to take a powerful, pretrained latent model (specifically the Diffusion Transformer or DiT) and swap out its "interface" to the world.
- Architecture: They keep the DiT backbone but discard the VAE. They replace the VAE encoder with a linear patch embedding and the VAE decoder with a lightweight, 3D convolutional "detailer head."
- Initialization: To avoid training from scratch, they use a linear map to approximate the VAE encoder. This "tricks" the DiT into thinking it's still seeing latents, allowing it to start training from a high-quality baseline.
- The Training Recipe: The authors identified three critical design choices to make pixel-space training work:
- $x$-prediction: Instead of predicting "velocity" (common in latent models), PixVid predicts the clean video directly. This prevents the model from trying to reconstruct high-frequency per-pixel noise, which leads to grainy results.
- Noise Shift: They use a larger noise shift during training because pixels are more redundant than latents and require more noise to "hide" the signal.
- Timestep Calibration: They mathematically align the signal-to-noise ratios between the original latent space and the new pixel space.
Datasets
The models were finetuned using:
- Text-to-Video: 64K landscape videos generated by Wan2.1.
- Video Editing: 79K stock videos with paired depth maps generated by "Video Depth Anything."
- Evaluation: The team used VBench for general quality, Movie Gen Video Bench for human preference studies, and various camera-path datasets (like ABot-World and DL3DV) to test interactive responsiveness.
Key Insights and Ideas
- Resolution for Free: In latent models, higher resolution means more tokens. In PixVid, you can simply double the patch size (e.g., from 16x16 to 32x32) and add one upsampling layer to the detailer head. This allows the model to generate $4\times$ more pixels using the exact same number of transformer tokens.
- Frame-wise is Faster: By moving to a frame-wise tokenization (where each token belongs to exactly one frame), the model can respond to user input at the very next frame. This is a game-changer for interactive "world models."
- Convolutional Detailers Matter: Using a simple linear head to turn tokens into pixels creates "checkerboard" artifacts. A convolutional head that looks at neighboring tokens smooths these boundaries out.
Key Results
- Quality: PixVid achieved a VBench total score of 84.78, slightly outperforming its latent baseline (Wan2.1-14B at 84.05).
- Latency: In interactive tests, PixVid reduced median control-to-screen latency from 495ms down to 34ms—a 14x improvement that makes real-time streaming possible.
- Motion Fidelity: In depth-conditioned editing, PixVid maintained 98% of the source's motion, whereas the latent model dropped to 83% due to temporal blurring.
- High-Res: The model successfully scaled to 960x1664 resolution while maintaining a token budget similar to a standard 480x832 latent model.
Limitations and Future Work
- Token Count at Low Res: Because PixVid doesn't compress time, it processes about 3.9x more tokens per second of video than a latent model at standard resolutions. While the gap closes at high resolutions, it makes short, low-res clips more expensive.
- Autoregressive Drift: Like most autoregressive models, PixVid can accumulate small errors over very long sequences, leading to "drift" in camera motion.
- New Training Frontiers: The authors note that generating directly in pixels opens the door for using pixel-level losses (like adversarial training or Reinforcement Learning with human feedback) that were previously difficult to apply through a frozen VAE.
Summary updated