TL;DR
T-GVC is a novel generative video coding framework that achieves high-quality video reconstruction at ultra-low bitrates by using sparse trajectory points to guide a pre-trained video diffusion model. By bridging low-level motion tracking with high-level semantic importance, it provides more precise motion control than text-based methods without requiring expensive model retraining.
What Problem Does This Paper Address?
In Ultra-Low Bitrate (ULB) scenarios—where bandwidth is extremely limited—traditional video codecs (like H.265/HEVC or H.266/VVC) and even standard neural codecs struggle. They often produce "blocky" artifacts or lose critical semantic information because they prioritize pixel-level fidelity over the "meaning" of the scene.
Generative Video Coding (GVC) has emerged as a solution, using AI to "re-imagine" the video based on compact cues. However, existing GVC methods have two major flaws:
- Domain Specificity: Many only work for specific content, like human faces or bodies.
- Imprecise Guidance: Methods that rely on text prompts or high-level sketches often fail to capture fine-grained motion, leading to "hallucinated" movements that look unrealistic or physically impossible.
Methods
The authors introduce T-GVC, which uses a three-pronged approach to solve these issues:
- Semantic-Aware Sparse Motion Sampling: Instead of sending dense optical flow (which is bitrate-heavy), the encoder identifies "motion instances" (objects moving together). It then uses a CLIP-based evaluation to determine which motions are semantically important and samples only a few key trajectory points for those areas.
- Training-Free Latent Space Guidance: This is the core innovation. Rather than training a new "ControlNet" or fine-tuning a massive model, T-GVC uses a pre-trained Video Diffusion Model (DynamiCrafter). During the denoising process (inference), it applies a "Motion Correction" gradient. This forces the latent features of the generated frames to align with the transmitted trajectories, ensuring the motion is physically plausible and faithful to the original.
- Variable-Length Generation: Since most diffusion models are trained on fixed-length clips (e.g., 16 frames), the authors developed a dual-stage generation strategy to handle longer video sequences while maintaining temporal consistency.
Datasets
The framework was evaluated on several standard video compression benchmarks:
- HEVC Class B & C: Standard sequences like BasketballDrill.
- UVG (Ultra Video Group): High-quality natural video sequences.
- MCL-JCV: A dataset specifically for video compression evaluation.
- Metrics: Performance was measured using LPIPS (perceptual quality), CLIP-SIM (semantic similarity), and DISTS, rather than traditional PSNR, which is poorly suited for generative tasks.
Key Insights and Ideas
- Motion as a Semantic Priority: The paper argues that not all motion is equal. By using CLIP to calculate an "importance score," the system spends its precious bit-budget on the movements that define the scene (like a player jumping) while ignoring noise or irrelevant background shifts.
- Geometry over Text: While many recent models use text prompts (e.g., "a man running"), T-GVC proves that geometric trajectories are a far more efficient and precise "language" for motion in a coding context.
- Zero-Shot Control: The ability to guide a diffusion model's latent space without retraining is a significant win for flexibility. It allows the system to leverage the powerful priors of state-of-the-art models (like DynamiCrafter) without the massive overhead of custom training for every new codec version.
Key Results
- Superior Efficiency: T-GVC consistently outperformed traditional codecs (H.265, H.266) and the state-of-the-art neural codec (DCVC-FM) in perceptual quality at bitrates below 0.005 bits per pixel (bpp).
- Ablation Success: In tests comparing text-only guidance vs. trajectory guidance, the trajectory-guided approach achieved a significantly better BD-rate (a measure of compression efficiency), showing that trajectories are essential for "content-faithful" reconstruction.
- Visual Fidelity: Qualitatively, T-GVC maintains the identity of small, fast-moving objects (like a basketball) that traditional codecs often blur into the background.
Limitations and Future Work
- Computational Complexity: Like most diffusion-based methods, T-GVC is computationally expensive. The dense trajectory tracking and the iterative denoising process mean it is currently far from real-time performance.
- Complex Transformations: While 2D trajectories work well for translation, the model still struggles with complex 3D transformations like rapid rotations or extreme occlusions, which can lead to minor visual artifacts.
- Encoder Optimization: The authors note that the image encoder (used for keyframes) could be further optimized to reach even lower bitrates while maintaining the structural integrity needed to seed the diffusion process.
Summary updated