TL;DR
ADE-GVC is the first "timestep-aware" generative video compression framework that adaptively selects when to start the diffusion denoising process based on a video's specific distortion level. By moving away from "one-size-fits-all" fixed entry points, it achieves a superior balance between reconstruction fidelity and perceptual realism, delivering state-of-the-art performance at extremely low bitrates.
What Problem Does This Paper Address?
Generative video compression, particularly using diffusion models, has become a leading paradigm for extreme low-bitrate scenarios. These models can "hallucinate" realistic textures where traditional codecs would simply produce a blurry mess. However, their effectiveness hinges on a delicate trade-off: fidelity vs. realism.
Current diffusion-based codecs typically start the denoising process from a compressed representation perturbed by a fixed amount of noise. This is problematic. If you inject too much noise, the model has too much "creative freedom" and loses fidelity to the original source. If you inject too little, the model can't effectively "clean up" the compression artifacts, leaving the video looking unrealistic. Because every video sequence degrades differently under compression, a fixed entry point is inherently sub-optimal.
Methods
The authors propose ADE-GVC (Adaptive Diffusion Entry for Generative Video Compression), which introduces three key components:
- Geometric Optimization Formulation: The researchers mathematically formulated the denoising entry selection as an optimization problem. Their analysis revealed a fundamental principle: the optimal entry timestep should decrease monotonically as compression distortion increases. In other words, more heavily distorted videos need an earlier entry into the denoising process (more noise injection) to be properly restored.
- Timestep Predictor (TP): A lightweight module that analyzes the compressed latent representation to estimate the "Goldilocks" denoising start point ($\tau$) for that specific sequence.
- Temporal Adaptor (TA): To ensure temporal consistency—a common headache in generative video—this module extracts compression-aware spatio-temporal features and injects them as conditions into the diffusion model.
The framework is built upon a pre-trained video diffusion backbone (Wan2.1-VACE) and uses a three-stage training strategy to align the compression network, the diffusion model, and the new adaptive predictor.
Datasets
- Training: A subset of the Pexels dataset, comprising approximately 480,000 high-quality video clips selected for aesthetic and clarity scores.
- Evaluation: Standard benchmarks including HEVC Class B, UVG, and MCL-JCV. These datasets provide a diverse range of motion and texture complexities to test the robustness of the codec.
Key Insights and Ideas
- The "Aha!" Correlation: The authors found a remarkably strong negative correlation (coefficient of -0.84) between their predicted start timesteps and the DISTS perceptual metric. This empirically validates their theoretical model: videos that the metric identifies as "more distorted" are exactly the ones the predictor chooses to denoise more aggressively.
- Distortion, Not Content: Interestingly, the predicted entry point is primarily driven by compression-induced distortion rather than the intrinsic characteristics of the source video (like brightness or color richness). This suggests the model is truly learning to "see" and react to the artifacts themselves.
- Efficiency through Adaptation: By shortening the generative refinement trajectory for low-distortion videos, the adaptive approach doesn't just improve quality—it avoids unnecessary computation.
Key Results
ADE-GVC sets a new bar for generative video compression:
- Massive Bitrate Savings: Compared to the latest open-source baseline (GLC-Video), ADE-GVC achieves average BD-rate savings of 40.99% on the DISTS perceptual metric and a staggering 95.24% on PSNR.
- Perceptual Superiority: It significantly outperforms existing methods like S²VC and GNVC-VD across all tested datasets in terms of LPIPS and DISTS.
- Visual Quality: Qualitatively, the method excels at recovering fine textures (like wall grains) and maintaining sharp, recognizable text boundaries that other generative methods often distort or hallucinate into nonsense.
- Speed: Through optimization strategies like parallel inference and BF16 precision, the authors achieved a 5.11x speedup, bringing decoding time down to ~0.83s per 1080p frame on an NVIDIA H20 GPU.
Limitations and Future Work
While ADE-GVC adaptively chooses where to start denoising, it currently uses a fixed number of sampling steps for every sequence. The authors acknowledge that a truly optimal system would likely adapt both the entry point and the number of steps to further boost efficiency.
Additionally, while this work focuses on video, the underlying theory of adaptive denoising entry is broadly applicable and could likely benefit generative image compression and other low-level vision tasks like super-resolution or dehazing.
Summary updated