Object-based rate allocation with spatio-temporal trade-offs

Jeong‐Woo Lee, Anthony Vetro, Yao Wang, Yo-Sung HoProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2002 · DOI

TL;DR

This paper introduces a novel rate control algorithm for MPEG-4 object-based video coding that optimizes bit allocation across multiple video objects (MVOs) by considering both spatial and temporal quality trade-offs. The key innovation is a method that dynamically selects coding modes and adjusts frame skipping to achieve a constant bit rate while significantly improving rate-distortion performance and preventing buffer overflow/underflow.

What Problem Does This Paper Address?

Traditional video coding rate control algorithms often struggle with the complexities introduced by object-based coding (MPEG-4), where video is segmented into arbitrarily shaped objects. A major challenge is allocating bits effectively to these individual objects while maintaining a consistent overall bit rate and ensuring smooth playback without buffer issues. Existing methods, even those adapted for object-based coding, don't adequately handle the dynamic nature of object-level bit allocation and the spatio-temporal quality trade-offs that arise. Specifically, the paper addresses the need for a rate control strategy that can:

  • Allocate bits at the object level: Different objects within a scene may have varying content complexity and importance, requiring differentiated bit allocation.
  • Manage spatio-temporal trade-offs: Decide whether to encode more frames at lower quality or fewer frames at higher quality, considering the impact on individual objects and the overall scene.
  • Prevent buffer overflow/underflow: Ensure that the encoded bitstream consistently fits within the decoder's buffer, a critical requirement for real-time streaming.
  • Improve Rate-Distortion (R-D) performance: Achieve better visual quality for a given bit rate compared to existing methods.

Methods

The paper proposes a rate control algorithm that extends frame-based rate control to the object-based domain by introducing several key components:

  • Distortion Models for Coded and Non-Coded VOs: The algorithm utilizes models to estimate the distortion of both encoded video objects (VOs) and those that are skipped (reconstructed by interpolation from previous frames). These models consider quantization parameters and temporal interpolation errors.
  • Spatio-Temporal Trade-off Models (R-D Coding Modes): The core of the novelty lies in defining and evaluating different coding modes (Case I and Case II) that balance coding a current frame versus skipping it to allocate more bits to a future frame. This decision is driven by minimizing distortion under buffer constraints.
  • Case I: Codes the current frame and the next frame.
  • Case II: Skips the current frame and allocates more bits to the next frame.
  • A further extension (Model II) addresses scenarios where the current Quantization Parameter (QP) is too high, introducing a "Case III" that involves skipping multiple frames to assign a lower QP to a later frame.
  • Rate Control Algorithm: A unified algorithm is presented that integrates these distortion models and coding modes. It aims to minimize distortion while adhering to overall bit rate and buffer occupancy constraints. The algorithm dynamically determines the optimal frame skip factor (f_s) and quantization parameter (QP) for each object.
  • Target Bit Calculation and Distribution: The paper details how target bits are calculated for a video object (VO) and then distributed among individual objects within that VO, considering factors like object size, motion, and variance. This ensures that each object receives an appropriate share of bits, even accounting for shape, motion, and header information.

Datasets

  • AKIYO sequence: This is the primary dataset used for evaluation. It's a CIF format sequence consisting of 2 video objects (VOs) and 300 frames, encoded at a full frame rate of 30 frames/s.

Key Insights and Ideas

  • Object-level R-D optimization is crucial: Simply extending frame-based rate control to objects isn't sufficient. The algorithm's success hinges on its ability to make independent R-D decisions for each object.
  • Dynamic frame skipping is key: Instead of fixed frame skipping, the proposed method intelligently skips frames based on a spatio-temporal trade-off analysis, aiming for minimum distortion under buffer constraints. This is a significant improvement over methods that only skip frames due to buffer limitations.
  • Buffer stability is achievable: A major takeaway is the algorithm's ability to maintain actual coded bits very close to target bits across a wide range of bit rates, leading to robust buffer occupancy and preventing overflow/underflow. This is a critical practical advantage.
  • Handling high QP scenarios: The introduction of Model II and Case III demonstrates a proactive approach to dealing with situations where a high QP might lead to poor quality, by strategically skipping frames to allow for better quality encoding later.

Key Results

  • Improved Rate-Distortion Performance: The proposed algorithm shows moderate improvements in R-D performance compared to the MPEG-4 reference rate control algorithm, particularly at low bit rates.
  • Accurate Bit Allocation: The ratio of actual coded bits to target bits is consistently close to 1 (within ±5% for the proposed method, compared to significant fluctuations in the reference method) across various bit rates (32 kbps to 256 kbps).
  • Buffer Stability: The algorithm successfully avoids buffer overflow and underflow, as demonstrated by stable buffer occupancy plots (around 50% with ±20% variation) across different bit rates. The reference method, in contrast, experienced overflows.

Limitations and Future Work

  • Frame-based Skipping: The paper notes that the proposed method supports only frame-based skipping for multiple video objects. While the distortion measure can be adapted for object-based skipping, this is presented as a consideration for future work.
  • Uniform Picture Quality Constraint: The paper mentions that the constraint of uniform picture quality from frame to frame can be difficult to achieve, especially when the current QP is strongly related to the previous QP. This suggests that further research could explore relaxing this constraint for potentially better R-D performance.
  • Standard of Judgement for QP: The paper acknowledges that choosing an exact threshold QP (e.g., 26) for deciding when a QP is "too large" is difficult and relies on simulation-based choices. Developing more adaptive or data-driven thresholds could be a future direction.

Summary updated

Abstract

This paper describes a bit allocation algorithm that can achieve a constant bit rate when coding multiple video objects (MVOs), while improving the rate-distortion (R-D) performance over the reference method for MPEG-4 object-based rate control. In object-based coding, bit allocation is performed at the object level and temporal rates of different objects may vary. The proposed algorithm in this paper deals with these two issues. We pay particular attention to maintenance of buffer occupancy levels and propose a new method for spatio-temporal trades-offs for object-based coding. In order to improve the coding efficiency, we also consider several new R-D coding modes. These modes are available at the encoder to determine the best R-D performance under different coding conditions and are chosen automatically. Simulation results demonstrate moderate improvements at low and high bit rates. One important key aspect of the proposed algorithm is that the actual coded bits in the proposed algorithm are similar to the target bits over a wide range of bit rates. Consequently, the proposed algorithm has not experienced the buffer overflow/underflow over the entire range of bit rates.
BibTeX
@inproceedings{lee2002object,
  author = {Jeong‐Woo Lee and Anthony Vetro and Yao Wang and Yo-Sung Ho},
  title = {{Object-based rate allocation with spatio-temporal trade-offs}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2002},
  doi = {10.1117/12.453078},
}
Figure