TL;DR
This paper identifies "latent capacity"—the number of compressed values stored per pixel—as the fundamental bottleneck preventing learned video codecs from reaching professional high-fidelity standards. By reintroducing encoder-side search for motion and residual partitions, the authors developed a learned codec that successfully spans the entire quality range from standard VVC test conditions up to production-grade ProRes HQ.
What Problem Does This Paper Address?
While learned video compression has made massive strides in recent years, it has mostly targeted "consumer" bitrates where some loss of detail is acceptable. Professional workflows—such as film acquisition, post-production, and archiving—operate at much higher fidelity levels, typically using 10-bit YCbCr 4:2:2 formats.
Existing learned codecs hit a "quality ceiling" (often around 38 dB PSNR) because they rely on feed-forward neural networks with a fixed, low latent capacity. Even if you remove quantization entirely, these models cannot represent the fine details required for high-end production. The authors define a "professional quality range" between the lowest standard test condition (VVC at QP 37) and the high-quality grade of the industry-standard ProRes codec to benchmark this gap.
Methods
The authors propose a hybrid coding loop that combines the strengths of neural networks with the exhaustive search strategies of traditional codecs.
Key technical innovations include:
- Encoder-Side Search: Unlike "one-pass" learned encoders that apply a fixed mapping, this codec searches for the best block-based motion and residual partitions for every frame, minimizing a rate-distortion cost ($J = D + \lambda R$).
- Variable Latent Capacity: The codec uses learned block transforms for sizes ranging from $4 \times 4$ to $64 \times 64$. Crucially, smaller blocks have a higher "latent capacity" (up to 1.0 latent values per sample for $4 \times 4$ blocks), allowing the model to allocate more "bandwidth" to complex textures.
- Learned Refinement: While the search is traditional, the prediction refinement, transforms, and in-loop filters are all deep neural networks trained to handle 10-bit 4:2:2 video natively.
Datasets
- Training: The model was trained on 2,590 clips from professional ProRes 4:2:2 sources and 160 sequences from the BVI-AOM dataset (converted to 4:2:2).
- Evaluation: Performance was measured on six standard 10-bit YCbCr 4:2:2 sequences from the JVET test set (e.g., Kimono1, Traffic, EBUHorse).
Key Insights and Ideas
- The Capacity Bottleneck: Most learned codecs are limited to a latent capacity of 0.25. The authors show that to reach professional fidelity, you need a capacity of at least 1.0. Without it, the network simply doesn't have enough "room" to store the necessary information, regardless of how fine the quantization is.
- Closing the Amortization Gap: Feed-forward encoders suffer from an "amortization gap"—they are optimized for the average case but fail on specific, difficult content. Per-block searching at encoding time is essential to bridge this gap for high-fidelity applications.
- Motion Search vs. Optical Flow: The paper finds that traditional block-based motion search handles distant reference frames significantly better than the optical flow methods typically favored in the learned compression literature.
Key Results
The proposed codec is the first learned model to successfully reach the upper end of the professional quality range (ProRes HQ) across all test sequences.
- Range Coverage: Previous state-of-the-art learned codecs only covered 24% to 38% of the professional quality range. The proposed adaptive codec covers 100%.
- Visual Fidelity: As shown in Figure 5, the adaptive partitioning allows the codec to capture fine structures like water streaks that fixed-capacity models blur out.
Limitations and Future Work
- Bitrate Efficiency: While it reaches high quality, the codec is currently less efficient than VVC and HEVC, requiring more bits to achieve the same PSNR.
- Computational Complexity: Encoding is slow, taking between 2 and 8.5 seconds per 1080p frame on a high-end GPU (RTX 6000).
- Incomplete System: The current evaluation focuses on inter-frame coding; intra frames (I-frames) are still provided by external sources. The authors aim to close the rate gap and integrate intra-coding in future work.
Summary updated