Studies on spatial scalable frameworks for motion aligned 3D wavelet video coding

Ruiqin Xiong, Jizheng Xu, Feng Wu, Shipeng Li2005 · DOI

TL;DR

This paper tackles the challenge of creating highly scalable video codecs that can adapt to diverse network conditions and device capabilities. The authors propose a novel "in-scale MATF" (ISMATF) framework that achieves good performance for fully scalable video coding by eliminating redundancy, outperforming existing redundant multi-T+2D approaches.

What Problem Does This Paper Address?

Streaming video over the internet is a complex task due to varying network bandwidths and client device capabilities (display resolution, processing power). To address this, scalable video coding aims to create a single bitstream that can be decoded at different quality levels (SNR scalability), resolutions (spatial scalability), or frame rates. This paper focuses on spatial scalability, specifically within the context of 3D wavelet video coding.

A key challenge in 3D wavelet video coding is the interaction between motion-aligned temporal filtering (MATF) and wavelet transforms. This interaction can break the independence of subbands, leading to issues like "SH drifting" (where information from higher resolution layers is lost at lower resolutions) and "operational mismatch" (where temporal filtering is applied at different resolutions for encoding and decoding). Existing approaches fall into three categories: T+2D (temporal filtering before spatial decomposition), 2D+T (spatial decomposition before temporal filtering within subbands), and multi-T+2D (temporal filtering applied at multiple spatial resolutions). The paper specifically investigates multi-T+2D schemes, which offer promising scalability but can suffer from redundancy introduced by representing frames at multiple resolutions. This redundancy can lead to significant performance degradation, especially for wide-range SNR scalability.

Methods

The paper compares three main types of spatial scalable video coding frameworks:

  • T+2D (Spatial-Domain MATF - SDMATF): Temporal filtering is applied to the full-resolution input frames before spatial decomposition. At the decoder, motion compensation is performed at a lower resolution, leading to operational mismatch and SH drifting.
  • 2D+T (In-Band MATF - IBMATF): Spatial decomposition is done first, and then temporal filtering is applied independently within each spatial subband. This avoids SH drifting but can be inefficient due to motion estimation and compensation in subbands. Overcomplete DWT (CODWT) is used to mitigate this.
  • Multi-T+2D: This category involves applying MATF at multiple spatial resolutions. The paper focuses on two variants:
  • Redundant Multi-T+2D (MRMATF): Frames are represented redundantly at multiple resolutions, and MATF is applied to each. This introduces cross-spatial layer redundancy, which the authors attempt to manage with inter-spatial-layer prediction. However, this redundancy leads to performance loss for wide-range scalability.
  • Non-Redundant Multi-T+2D (ISMATF - In-Scale MATF): This is the novel contribution. It also uses multiple MATF operations but does so in a "scale-by-scale" manner, processing subbands within the current resolution only. This ensures a non-redundant representation and avoids SH drifting, aiming for better performance in fully scalable video coding.

The authors implement and compare these frameworks on their codec platform, using a consistent motion vector field (MVF) obtained from full-resolution motion search for all schemes to ensure a fair comparison.

Datasets

The paper uses standard MPEG test sequences for evaluation: Foreman, Mobile, Football, and Coastguard. These sequences are used at both QCIF (15Hz) and CIF (30Hz) resolutions. No new datasets are introduced.

Key Insights and Ideas

  • Redundancy is a double-edged sword: While redundant representations in multi-T+2D schemes can simplify scalability, the inherent redundancy significantly degrades compression efficiency, especially for wide-range SNR scalability.
  • Inter-layer prediction helps, but not enough: For redundant multi-T+2D schemes, inter-spatial-layer prediction can improve performance for narrow-range SNR scalability by exploiting correlations between layers. However, it doesn't fully overcome the overhead and limitations of the redundant representation for wide-range scalability.
  • "In-scale" MATF is key: The proposed ISMATF framework's core innovation is applying MATF "in-scale," meaning it only uses subbands within the current resolution. This elegantly eliminates redundancy and SH drifting, leading to superior performance for fully scalable video coding.
  • Trade-offs in spatial scalability: SDMATF offers good high-resolution performance but suffers at low resolutions due to mismatch. IBMATF provides scalability but at the cost of compression efficiency. Multi-T+2D schemes aim for a balance, but the redundant approach (MRMATF) has limitations.
  • Shift-variance impact: The ISMATF, while efficient, can incur a slight performance loss compared to SDMATF due to not utilizing all SH information from higher resolutions, leading to some shift-variance issues, particularly noticeable in sequences with high-frequency textures like "Mobile."

Key Results

  • Low Resolution Performance: ISMATF and MRMATF perform very well, often superior to SDMATF with optimal rate allocation, with no mismatch or drifting. SDMATF's baseline performance is poor due to operational mismatch and SH drifting.
  • High Resolution Performance: SDMATF shows the best performance due to efficient full-resolution MC. MRMATF with unlimited SNR scalability performs poorly due to redundancy.
  • ISMATF vs. MRMATF: ISMATF is superior to MRMATF for its non-redundancy, showing a performance gain of up to 1.3dB at high bit rates.
  • MRMATF with Early Truncation: For limited SNR scalability, MRMATF can be improved by early truncation, approaching SDMATF's performance at high bit rates. This makes it practical for narrow-range scalability.
  • Energy in SL Bands: The paper quantifies the energy increase in SL bands when SH subbands are not exploited in ISMATF, showing a significant increase for sequences with high-frequency textures.

Limitations and Future Work

  • ISMATF's Shift-Variance: The paper acknowledges that ISMATF's performance loss compared to SDMATF is due to not utilizing all SH information, leading to shift-variance issues, especially in high-frequency content.
  • Future Work: The authors suggest exploring a combined lifting signal for ISMATF, aiming to improve MC efficiency by incorporating information from other resolutions, while carefully managing the trade-off with potential SH drifting. They also note that for multi-T+2D schemes, additional motion information can be beneficial for high-resolution layers, particularly in redundant schemes like MRMATF with early truncation.

Summary updated

Abstract

This paper makes a comparative study on the various spatial scalable coding frameworks. The frameworks with multiple image-domain motion aligned temporal filtering at various spatial resolutions, named as multi-T+2D, are mainly investigated. First we investigate a multi-T+2D scheme based on redundant frame representation. The cross spatial layer redundancy and prediction methods are discussed. The redundancy brings significant performance loss for schemes providing wide range SNR spatial scalability. To remove the redundancy produced in the multi-resolution temporal filtering while retaining the advantage of spatial-domain motion compensation, a novel non-redundant multi-T+2D scheme is proposed. Performance comparison is given among the discussed frameworks and it shows that the proposed non-redundant multi-T+2D framework has a good performance for fully scalable video coding. We also verify that the redundant multi-T+2D framework with cross spatial layer reconstruction feedback is practical in providing narrow range SNR scalability for each spatial layer.
BibTeX
@misc{xiong2005studies,
  author = {Ruiqin Xiong and Jizheng Xu and Feng Wu and Shipeng Li},
  title = {{Studies on spatial scalable frameworks for motion aligned 3D wavelet video coding}},
  year = {2005},
  doi = {10.1117/12.631549},
}
Figure