Abstract
Recent advances in video generation techniques have given rise to an emerging paradigm of generative video coding for Ultra-Low Bitrate (ULB) scenarios by leveraging powerful generative priors. However, most existing methods are limited by domain specificity (e.g., facial or human videos) or excessive dependence on high-level text guidance, which tend to inadequately capture fine-grained motion details, leading to unrealistic or incoherent reconstructions. To address these challenges, we propose Trajectory-Guided Generative Video Coding (dubbed T-GVC), a novel framework that bridges low-level motion tracking with high-level semantic understanding. T-GVC features a semantic-aware sparse motion sampling pipeline that extracts pixel-wise motion as sparse trajectory points based on their semantic importance, significantly reducing the bitrate while preserving critical temporal semantic information. In addition, by integrating trajectory-aligned loss constraints into diffusion processes, we introduce a training-free guidance mechanism in latent space to ensure physically plausible motion patterns without sacrificing the inherent capabilities of generative models. Experimental results demonstrate that T-GVC outperforms both traditional and neural video codecs under ULB conditions. Furthermore, additional experiments confirm that our framework achieves more precise motion control than existing text-guided methods, paving the way for a novel direction of generative video coding guided by geometric motion modeling.
Introduction
In Ultra-Low Bitrate (ULB) scenarios, where bandwidth is extremely limited, traditional video codecs like H.265/HEVC and H.266/VVC often struggle, producing significant artifacts or losing semantic meaning. Generative Video Coding (GVC) addresses this by using AI to reconstruct scenes from compact cues. However, current GVC approaches often suffer from domain specificity or imprecise guidance from text prompts, leading to physically implausible "hallucinated" motions.
T-GVC (Trajectory-Guided Generative Video Coding) bridges the gap between low-level motion tracking and high-level semantics. By utilizing sparse trajectory points instead of dense flow or vague text, it achieves precise motion control at ultra-low bitrates without requiring expensive model retraining.
Methodology
The T-GVC framework consists of three primary components: a semantic-aware sparse motion sampler, a training-free latent space guidance mechanism, and a variable-length generation strategy.
Semantic-Aware Sparse Motion Sampling
Rather than transmitting dense optical flow, the encoder identifies "motion instances"—groups of pixels moving coherently. It employs a CLIP-based evaluation to assign "semantic importance" scores to these motions. Only the most semantically relevant trajectory points (e.g., a player's jump in a basketball game) are sampled and transmitted, effectively prioritizing the "meaning" of the scene over background noise.
Training-Free Latent Space Guidance
T-GVC leverages a pre-trained Video Diffusion Model (DynamiCrafter) without requiring fine-tuning. During the denoising process, a "Motion Correction" gradient is applied in the latent space. This mechanism forces the generated frames to align with the transmitted sparse trajectories, ensuring that the reconstructed motion is both physically plausible and faithful to the original source.
Experimental Results
The framework was evaluated on standard benchmarks including HEVC Class B & C, UVG, and MCL-JCV. Performance was assessed using perceptual and semantic metrics such as LPIPS, CLIP-SIM, and DISTS, which are better suited for generative tasks than traditional PSNR.
T-GVC consistently outperformed traditional codecs (H.265, H.266) and state-of-the-art neural codecs (DCVC-FM) at bitrates below 0.005 bits per pixel (bpp). Ablation studies confirmed that trajectory guidance is significantly more efficient than text-only guidance, achieving a superior BD-rate and maintaining the visual identity of small, fast-moving objects that traditional codecs often blur.
Conclusion
T-GVC demonstrates that geometric trajectories provide a more precise and efficient language for motion in generative video coding than high-level text. While the current implementation faces challenges in computational complexity and complex 3D transformations, it establishes a promising new direction for content-faithful video reconstruction at ultra-low bitrates. Future work will focus on optimizing the encoder and improving real-time performance.
Summary updated