VideoMAETok: Boosting Video Diffusion Models via Masked Autoencoders as Tokenizers

Zhan Tong, Tinne TuytelaarsICML 2026 · OpenReview

Latent Diffusion Models (LDMs) have become the dominant paradigm for video generation. Instead of operating on raw pixels, these models rely on a tokenizer—an autoencoder that compresses video into a compact "latent space" where the generative learning actually takes place. However, a fundamental mismatch exists in how these tokenizers are traditionally trained versus how they are used in practice.

Most video tokenizers are optimized purely for reconstruction fidelity; their goal is to make the output look exactly like the input. Diffusion models, on the other hand, spend their time denoising heavily corrupted data. This creates a "training-inference gap": the tokenizer is optimized for clean data, but the generator is forced to navigate a noisy, uncertain latent space. Surprisingly, a tokenizer that achieves near-perfect reconstruction can actually be a terrible partner for a diffusion model, leading to poor generation quality.

To bridge this gap, researchers introduce VideoMAETok, a new family of Vision Transformer (ViT)-based video tokenizers designed specifically for latent diffusion models.

Rethinking Tokenization: From Reconstruction to Corruption-Inversion

VideoMAETok shifts the training objective from simple reconstruction to corruption-inversion. Its architecture and training strategy are built on three key pillars:

  1. Masked Modeling for Efficiency: Borrowing from Masked Autoencoders (MAE), the model applies a high-ratio random mask (up to 80-90%) to the video tokens. The encoder only processes the visible, unmasked tokens, making the process highly computationally efficient.
  2. Latent Denoising: During training, interpolative Gaussian noise is injected into the latent features. This forces the decoder to learn how to recover clean signals from the exact kind of "messy" data a diffusion model produces, encouraging latents that remain informative and well-conditioned for downstream denoising.
  3. Asymmetric Architecture: Unlike standard VideoMAE, which uses a heavy encoder for recognition tasks, VideoMAETok employs a lightweight encoder and a heavy decoder. This design allocates more computational power to the complex task of mapping compressed latents back into high-fidelity pixels.

Key Insights: The Reconstruction Paradox

One of the paper's most striking findings is the Reconstruction Paradox. The authors demonstrate that lowering the reconstruction error (measured by rFVD) too much can actually cause the generation quality (gFVD) to deteriorate. Prioritizing pixel-perfect reconstruction often results in a latent space that is too fragile and entangled for diffusion models to navigate effectively.

By contrast, corruption-aware training produces a much more structured and well-spread latent manifold. Furthermore, the authors found that "non-causal" attention—allowing the model to look at both past and future frames—is superior for tokenization. While causal attention is necessary for streaming applications, it restricts the tokenizer's ability to compress video by exploiting bidirectional temporal context.

State-of-the-Art Results and Efficiency

VideoMAETok was rigorously evaluated across benchmark datasets, including Kinetics-400, Kinetics-600, UCF-101, and VBench. The results highlight significant improvements in both quality and efficiency:

  • Unprecedented Generation Quality: When paired with off-the-shelf generators like SiT-XL, VideoMAETok achieved a generation Fréchet Video Distance (gFVD) of 42.16 on Kinetics-600, obliterating the previous best baseline (VFRTok-L at 124.78).
  • Computational Efficiency: Because the encoder only processes unmasked tokens, VideoMAETok is an order of magnitude faster than competitors. For instance, it requires only 0.78 TFLOPs compared to Cosmos-L's 5.82 TFLOPs, while delivering superior results.
  • Qualitative Excellence: Generated samples exhibit significantly better temporal consistency, smoother motion, and superior preservation of fine details (such as limb articulation) compared to traditional VAE-based approaches.
  • Scalability: The method scales predictably; increasing the size of either the tokenizer or the generator consistently yields better video quality.

Limitations and Future Directions

While VideoMAETok sets a new standard, it introduces a fidelity trade-off. Training under heavy corruption inherently makes the raw reconstruction slightly worse. The authors suggest that "decoder fine-tuning" on clean data can mitigate this, but finding a single training objective that perfectly balances reconstruction and generation remains an open challenge.

Future work may also explore semantic alignment. The current model focuses on low-level pixel reconstruction, but incorporating semantic distillation using features from models like CLIP or DINOv2 could further enhance the high-level meaning of the learned tokens. Finally, while the model handles 16-32 frame clips exceptionally well, scaling to very long-form video generation will require architectural tweaks to manage memory and long-range temporal dependencies.

Summary updated

Abstract

Latent diffusion models have become the dominant paradigm for video generation, making the video tokenizer a critical role. While most existing tokenizers are trained primarily for reconstruction, diffusion models are optimized to denoise heavily corrupted latents, which creates a mismatch between tokenizer training objectives and downstream generative learning. As a result, reconstruction metrics (e.g., rFVD) can be a poor proxy for generation quality (gFVD), and overly prioritizing reconstruction may even hinder diffusion training. We propose VideoMAETok, a simple family of ViT-based video tokenizers trained explicitly as corruption-inversion models for latent video diffusion. VideoMAETok builds on masked autoencoders: we (i) apply high-ratio token masking and encode only visible spatiotemporal tokens for efficiency, and (ii) corrupt latent tokens with interpolative Gaussian noise to better match the denoising nature of diffusion generators. Training under such corruption encourages latents that remain informative and well-conditioned for downstream denoising. Extensive experiments show that VideoMAETok consistently improves generation quality when paired with off-the-shelf diffusion models (SiT and LightningDiT), achieving state-of-the-art gFVD on Kinetics-600 and UCF-101 while remaining compute-efficient. Code is available at https://github.com/yztongzhan/VideoMAETok.
BibTeX
@inproceedings{tong2026videomaetok,
  author = {Zhan Tong and Tinne Tuytelaars},
  title = {{VideoMAETok: Boosting Video Diffusion Models via Masked Autoencoders as Tokenizers}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=uLTgWANmgz},
}
Figure