TL;DR
This paper introduces a novel "look-ahead and feedback" (LFAM) model for joint rate allocation in High Efficiency Video Coding (HEVC). By incorporating feedback on actual distortion and bitrate alongside predicted complexity, LFAM significantly reduces visual quality variance across multiple video streams compared to existing methods, achieving an average saving of 65.94% in Mean Squared Error (MSE) variance.
What Problem Does This Paper Address?
In multimedia applications, multiple video streams are often transmitted over a limited bandwidth. A key challenge is how to allocate this bandwidth effectively to ensure a good user experience. Simply encoding each stream independently can lead to significant variations in visual quality. To address this, statistical multiplexing is used, where bandwidth is shared. Existing methods primarily focus on two objectives: minimizing average distortion (minAVE) to maximize overall visual quality, or minimizing distortion variance (minVAR) to ensure consistent quality across all streams. This paper focuses on the minVAR problem, aiming for equal visual quality. Traditional minVAR approaches rely on "look-ahead allocation models" (LAM) that assign bits based on predicted video complexity. However, the performance of these LAMs is highly sensitive to the accuracy of the complexity prediction, which can be difficult to achieve perfectly. This paper seeks to improve upon LAM by developing a more robust allocation strategy that is less dependent on perfect complexity prediction.
Methods
The core contribution is the proposed Look-ahead and Feedback Allocation Model (LFAM). Unlike traditional LAMs that solely rely on predicted complexity, LFAM integrates three pieces of information:
- Look-ahead Complexity Measures: These are predictions of video complexity for upcoming frames or groups of frames (super GOPs), similar to existing LAMs.
- Feedback Bitrate: The actual bitrate used for previously encoded frames.
- Feedback Distortion: The actual distortion (measured as Mean Squared Error - MSE) of previously encoded frames.
The LFAM model is theoretically derived from the minVAR objective function. It uses the feedback information to adjust bit allocation for the next super GOP, making it more adaptive to the actual coding performance rather than just predictions. The authors integrate this LFAM model into the HEVC reference software (HM16.0) and test it with various complexity measures previously proposed in the literature ([6]-[10]).
Datasets
The paper uses test sequences recommended by JCT-VC (Joint Collaborative Team on Video Coding). These sequences are divided into six classes (A through F) for evaluation. Specific details on the number of frames tested for each class are provided, indicating a comprehensive evaluation across different types of video content. No new datasets were introduced.
Key Insights and Ideas
- Feedback is Crucial for Robustness: The most significant insight is that incorporating actual bitrate and distortion feedback from the encoder dramatically improves rate allocation performance, making it less reliant on perfect complexity prediction. This is a key departure from prior LAM approaches.
- LFAM Adapts to Real-World Coding: By using feedback, LFAM can compensate for inaccuracies in complexity prediction and adapt to the dynamic nature of video coding, leading to more stable and consistent quality.
- Negligible Computational Cost: The proposed LFAM model is designed to require minimal additional computational resources, making it practical for real-time video coding applications.
- Universality of Complexity Measures: The LFAM model can work with various existing complexity measures, demonstrating its flexibility and ability to leverage prior research.
Key Results
- Significant Variance Reduction: The LFAM model achieved an average saving of 65.94% in MSE variance across all tested complexity measures and stream sets compared to the LAM approach.
- Improved Quality Consistency: Experimental results (Tables 2 and 3, Figure 2) show that LFAM significantly reduces the variance of MSE across super GOPs, leading to more uniform visual quality among different video streams.
- Better Performance Across Classes: LFAM consistently outperformed LAM across all tested classes (A-F) and with all evaluated complexity measures, demonstrating its general applicability.
- PSNR Improvement: In some cases, LFAM improved the PSNR of lower-quality sequences at the expense of slightly reducing the PSNR of higher-quality sequences, resulting in a more balanced and consistent quality distribution.
Limitations and Future Work
- Complexity Measure Dependence (Partial): While LFAM is less dependent on complexity measures than LAM, the paper acknowledges that the initial choice of complexity measure can still influence performance. The authors state that the comparison between different complexity measures themselves was not entirely fair due to potential variations in how authors in prior works set constant values.
- Future Work: The paper suggests that the proposed LFAM is a better solution for supplementing the inaccuracy of complexity measures with minimal computational cost. Future work could explore optimizing the feedback mechanism or further refining the complexity prediction alongside the LFAM framework. The paper also notes that the accuracy of the model is not fully dependent on complexity measure accuracy, implying that further research could explore this relationship more deeply.
Summary updated