Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park, Aryan V S, Dennis Yuren Menn, Ming-Hsuan Yang, Xun HuangarXiv 2026 · arXiv

TL;DR

PixVid is a simple yet powerful recipe for "lifting" pretrained latent video diffusion models directly into pixel space. By replacing the restrictive VAE with a frame-wise convolutional head, PixVid enables $2\times$ higher resolution at no extra token cost, slashes interactive latency from 495ms to 34ms, and eliminates the motion blur common in temporally compressed latent models.

What Problem Does This Paper Address?

Most state-of-the-art video generation models (like Sora or Wan2.1) don't work with pixels directly. Instead, they use a Variational Autoencoder (VAE) to compress video into a "latent space." While this makes training computationally feasible, it introduces three major bottlenecks:

  1. The Resolution Tax: Doubling the resolution quadruples the number of latent tokens, making high-definition video exponentially expensive to generate.
  2. Temporal Lag: Many VAEs compress time, meaning several frames are squashed into a single latent. In interactive applications (like games), the model can't respond to a user's input until an entire "block" of frames is ready, leading to noticeable lag.
  3. Motion Artifacts: Because multiple frames share one latent representation, fast-moving objects often appear blurred or "morphed" as the model struggles to represent distinct positions within a single compressed token.

The authors of PixVid argue that we should go "back to pixels" to overcome these structural limitations.

Methods

The core idea of PixVid is to take a powerful, pretrained latent model (specifically the Diffusion Transformer or DiT) and swap out its "interface" to the world.

  • Architecture: They keep the DiT backbone but discard the VAE. They replace the VAE encoder with a linear patch embedding and the VAE decoder with a lightweight, 3D convolutional "detailer head."
  • Initialization: To avoid training from scratch, they use a linear map to approximate the VAE encoder. This "tricks" the DiT into thinking it's still seeing latents, allowing it to start training from a high-quality baseline.
  • The Training Recipe: The authors identified three critical design choices to make pixel-space training work:
  • $x$-prediction: Instead of predicting "velocity" (common in latent models), PixVid predicts the clean video directly. This prevents the model from trying to reconstruct high-frequency per-pixel noise, which leads to grainy results.
  • Noise Shift: They use a larger noise shift during training because pixels are more redundant than latents and require more noise to "hide" the signal.
  • Timestep Calibration: They mathematically align the signal-to-noise ratios between the original latent space and the new pixel space.
Lifting a pretrained latent video DiT to pixel space.

Datasets

The models were finetuned using:

  • Text-to-Video: 64K landscape videos generated by Wan2.1.
  • Video Editing: 79K stock videos with paired depth maps generated by "Video Depth Anything."
  • Evaluation: The team used VBench for general quality, Movie Gen Video Bench for human preference studies, and various camera-path datasets (like ABot-World and DL3DV) to test interactive responsiveness.

Key Insights and Ideas

  • Resolution for Free: In latent models, higher resolution means more tokens. In PixVid, you can simply double the patch size (e.g., from 16x16 to 32x32) and add one upsampling layer to the detailer head. This allows the model to generate $4\times$ more pixels using the exact same number of transformer tokens.
  • Frame-wise is Faster: By moving to a frame-wise tokenization (where each token belongs to exactly one frame), the model can respond to user input at the very next frame. This is a game-changer for interactive "world models."
  • Convolutional Detailers Matter: Using a simple linear head to turn tokens into pixels creates "checkerboard" artifacts. A convolutional head that looks at neighboring tokens smooths these boundaries out.
Comparison of control-to-screen latency between latent and pixel models.

Key Results

  • Quality: PixVid achieved a VBench total score of 84.78, slightly outperforming its latent baseline (Wan2.1-14B at 84.05).
  • Latency: In interactive tests, PixVid reduced median control-to-screen latency from 495ms down to 34ms—a 14x improvement that makes real-time streaming possible.
  • Motion Fidelity: In depth-conditioned editing, PixVid maintained 98% of the source's motion, whereas the latent model dropped to 83% due to temporal blurring.
  • High-Res: The model successfully scaled to 960x1664 resolution while maintaining a token budget similar to a standard 480x832 latent model.
Depth-conditioned editing comparison showing PixVid's superior motion handling.

Limitations and Future Work

  • Token Count at Low Res: Because PixVid doesn't compress time, it processes about 3.9x more tokens per second of video than a latent model at standard resolutions. While the gap closes at high resolutions, it makes short, low-res clips more expensive.
  • Autoregressive Drift: Like most autoregressive models, PixVid can accumulate small errors over very long sequences, leading to "drift" in camera motion.
  • New Training Frontiers: The authors note that generating directly in pixels opens the door for using pixel-level losses (like adversarial training or Reinforcement Learning with human feedback) that were previously difficult to apply through a frozen VAE.

Summary updated

Abstract

Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation -- doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block. We present PixVid, a simple yet effective method that lifts a pretrained latent video model to directly generate every frame in pixel space. PixVid keeps the diffusion transformer and replaces the VAE decoder with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality. Finetuned from Wan2.1-14B, PixVid is competitive with the latent baseline within $\sim 1.3$M training clips. Working framewise in pixels, it generates at $2\times$ higher resolution with the same per-frame token count, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame. Project page: https://chenshuo20.github.io/PixVid/
BibTeX
@misc{song2026back,
  author = {Yanke Song and Shuo Chen and Chris Rockwell and Guande He and Nan Liu and Desai Xie and Tianpei Gu and Yong-Hyun Park and Aryan V S and Dennis Yuren Menn and Ming-Hsuan Yang and Xun Huang},
  title = {{Back to Pixels: Lifting Latent Video Diffusion into Pixel Space}},
  year = {2026},
  eprint = {2611.17336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17336},
}
Figure