Abstract
Neural video compression algorithms are nearly competitive with hand-crafted codecs in terms of rate-distortion performance and subjective quality. However, many neural codecs are inflexible black boxes, and give users little to no control over the reconstruction quality and bitrate. In this work, we present a flexible neural video codec that combines ideas from variable-bitrate codecs and region-of-interest-based coding. By conditioning our model on a global rate-distortion tradeoff parameter and a region-of-interest (ROI) mask, we obtain dynamic control over the per-frame bitrate and the reconstruction quality in the ROI at test time. The resulting codec enables practical use cases such as coding under bitrate constraints with fixed ROI quality, while taking a negligible hit in performance compared to a fixed-rate model. We find that our codec performs best on sequences with complex motion, where we substantially outperform non-ROI codecs in the region of interest with Bjøntegaard-Delta rate savings exceeding 60%.
Introduction
Traditional video codecs like H.264 and H.265 offer fine-grained control over bitrate and quality. In contrast, state-of-the-art neural video codecs have historically been "inflexible black boxes," where a single model is trained for one specific bitrate. Supporting multiple bitrates typically requires deploying multiple memory-intensive models. Furthermore, most neural codecs treat every pixel with equal importance, failing to prioritize the subject of a video (the Region of Interest, or ROI) over the background. This research introduces a unified neural codec that allows for simultaneous, dynamic control over both global bitrate and local ROI quality at test time.
Methods
The proposed architecture builds upon the Scale-Space Flow (SSF) framework, modified to incorporate two primary control parameters:
- Global Control ($\beta$): This parameter dictates the overall tradeoff between bitrate and distortion for the entire frame. The model utilizes "condition blocks"—conditional convolutions that scale and shift feature maps based on a one-hot embedding of $\beta$.
- Local Control ($\alpha$): This parameter determines the prioritization of the ROI relative to the background.
The loss function is redesigned to balance these priorities: $\mathcal{L} = \mathbb{E} [\beta \mathcal{L}_{rate} + \mathcal{L}_{dist}^{ROI} + \frac{1}{\alpha} \mathcal{L}_{dist}^{BG}]$. At inference, a user provides a binary ROI mask and selects values for $\alpha$ and $\beta$. These are combined into a weighted ROI mask fed into the encoders, allowing the model to dynamically shift bits from the background to the foreground without retraining.
Experimental Results
The model was trained on the Vimeo90K dataset using synthetic masks generated via Perlin noise to teach ROI awareness. Evaluation was conducted on DAVIS (real-world ROIs), UVG, and HEVC Class E2 (using pseudo-ground truth masks from an HRNet segmentation model).
Performance Gains
The codec achieved significant Bjøntegaard-Delta (BD) rate savings exceeding 60% in the ROI compared to non-ROI baselines. Perceptual metrics such as VMAF and NIQE also showed consistent improvements, confirming that prioritizing bits for salient regions significantly boosts perceived quality.
Key Insights
- Motion Complexity: ROI coding is most effective in videos with complex motion. In sequences with static backgrounds, standard motion compensation is already highly efficient, reducing the relative benefit of ROI prioritization.
- Temporal Propagation: The researchers found that low-quality backgrounds in initial I-frames can cause blurriness to propagate through subsequent P-frames. A strategy of high-quality I-frame transmission was found to mitigate this.
- Generalization: Training on synthetic Perlin noise masks generalizes effectively to real-world semantic objects like people or vehicles.
Conclusion and Future Work
This work demonstrates that a single neural network can navigate the continuous spectrum of bitrate and spatial importance. While highly effective, the model shows moderate sensitivity to the quality of the ROI mask; inaccurate masks can lead to faster degradation of non-ROI quality. Future research will focus on improving temporal consistency for rapidly changing ROIs and integrating these conditioning mechanisms into more advanced base architectures.
Summary updated