T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates

Zhitao Wang, Hengyu Man, Wenrui Li, Xingtao Wang, Xiaopeng Fan, Debin Zhao

TL;DR

T-GVC is a novel generative video coding framework that achieves high-quality video reconstruction at ultra-low bitrates by using sparse trajectory points to guide a pre-trained video diffusion model. By bridging low-level motion tracking with high-level semantic importance, it provides more precise motion control than text-based methods without requiring expensive model retraining.

What Problem Does This Paper Address?

In Ultra-Low Bitrate (ULB) scenarios—where bandwidth is extremely limited—traditional video codecs (like H.265/HEVC or H.266/VVC) and even standard neural codecs struggle. They often produce "blocky" artifacts or lose critical semantic information because they prioritize pixel-level fidelity over the "meaning" of the scene.

Generative Video Coding (GVC) has emerged as a solution, using AI to "re-imagine" the video based on compact cues. However, existing GVC methods have two major flaws:

  1. Domain Specificity: Many only work for specific content, like human faces or bodies.
  2. Imprecise Guidance: Methods that rely on text prompts or high-level sketches often fail to capture fine-grained motion, leading to "hallucinated" movements that look unrealistic or physically impossible.

Methods

The authors introduce T-GVC, which uses a three-pronged approach to solve these issues:

  • Semantic-Aware Sparse Motion Sampling: Instead of sending dense optical flow (which is bitrate-heavy), the encoder identifies "motion instances" (objects moving together). It then uses a CLIP-based evaluation to determine which motions are semantically important and samples only a few key trajectory points for those areas.
  • Training-Free Latent Space Guidance: This is the core innovation. Rather than training a new "ControlNet" or fine-tuning a massive model, T-GVC uses a pre-trained Video Diffusion Model (DynamiCrafter). During the denoising process (inference), it applies a "Motion Correction" gradient. This forces the latent features of the generated frames to align with the transmitted trajectories, ensuring the motion is physically plausible and faithful to the original.
  • Variable-Length Generation: Since most diffusion models are trained on fixed-length clips (e.g., 16 frames), the authors developed a dual-stage generation strategy to handle longer video sequences while maintaining temporal consistency.

Datasets

The framework was evaluated on several standard video compression benchmarks:

  • HEVC Class B & C: Standard sequences like BasketballDrill.
  • UVG (Ultra Video Group): High-quality natural video sequences.
  • MCL-JCV: A dataset specifically for video compression evaluation.
  • Metrics: Performance was measured using LPIPS (perceptual quality), CLIP-SIM (semantic similarity), and DISTS, rather than traditional PSNR, which is poorly suited for generative tasks.

Key Insights and Ideas

  • Motion as a Semantic Priority: The paper argues that not all motion is equal. By using CLIP to calculate an "importance score," the system spends its precious bit-budget on the movements that define the scene (like a player jumping) while ignoring noise or irrelevant background shifts.
  • Geometry over Text: While many recent models use text prompts (e.g., "a man running"), T-GVC proves that geometric trajectories are a far more efficient and precise "language" for motion in a coding context.
  • Zero-Shot Control: The ability to guide a diffusion model's latent space without retraining is a significant win for flexibility. It allows the system to leverage the powerful priors of state-of-the-art models (like DynamiCrafter) without the massive overhead of custom training for every new codec version.

Key Results

  • Superior Efficiency: T-GVC consistently outperformed traditional codecs (H.265, H.266) and the state-of-the-art neural codec (DCVC-FM) in perceptual quality at bitrates below 0.005 bits per pixel (bpp).
  • Ablation Success: In tests comparing text-only guidance vs. trajectory guidance, the trajectory-guided approach achieved a significantly better BD-rate (a measure of compression efficiency), showing that trajectories are essential for "content-faithful" reconstruction.
  • Visual Fidelity: Qualitatively, T-GVC maintains the identity of small, fast-moving objects (like a basketball) that traditional codecs often blur into the background.

Limitations and Future Work

  • Computational Complexity: Like most diffusion-based methods, T-GVC is computationally expensive. The dense trajectory tracking and the iterative denoising process mean it is currently far from real-time performance.
  • Complex Transformations: While 2D trajectories work well for translation, the model still struggles with complex 3D transformations like rapid rotations or extreme occlusions, which can lead to minor visual artifacts.
  • Encoder Optimization: The authors note that the image encoder (used for keyframes) could be further optimized to reach even lower bitrates while maintaining the structural integrity needed to seed the diffusion process.

Summary updated

Abstract

Recent advances in video generation techniques have given rise to an emerging paradigm of generative video coding for Ultra-Low Bitrate (ULB) scenarios by leveraging powerful generative priors. However, most existing methods are limited by domain specificity (e.g., facial or human videos) or excessive dependence on high-level text guidance, which tend to inadequately capture fine-grained motion details, leading to unrealistic or incoherent reconstructions. To address these challenges, we propose Trajectory-Guided Generative Video Coding (dubbed T-GVC), a novel framework that bridges low-level motion tracking with high-level semantic understanding. T-GVC features a semantic-aware sparse motion sampling pipeline that extracts pixel-wise motion as sparse trajectory points based on their semantic importance, significantly reducing the bitrate while preserving critical temporal semantic information. In addition, by integrating trajectory-aligned loss constraints into diffusion processes, we introduce a training-free guidance mechanism in latent space to ensure physically plausible motion patterns without sacrificing the inherent capabilities of generative models. Experimental results demonstrate that T-GVC outperforms both traditional and neural video codecs under ULB conditions. Furthermore, additional experiments confirm that our framework achieves more precise motion control than existing text-guided methods, paving the way for a novel direction of generative video coding guided by geometric motion modeling.
BibTeX
@inproceedings{wang2026t,
  author = {Zhitao Wang and Hengyu Man and Wenrui Li and Xingtao Wang and Xiaopeng Fan and Debin Zhao},
  title = {{T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i12.38014},
}

Loading related content...

Figure