Low-complexity Bayer-pattern video compression using distributed video coding

Hu Chen, Mingzhe Sun, Eckehard SteinbachProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2008 · DOI

TL;DR

This paper introduces a novel method for compressing video from single-chip cameras (which capture Bayer-patterned images) by optimizing the color space conversion and subsampling process. This approach significantly reduces encoding complexity (by nearly 50%) and improves video quality, especially when combined with distributed video coding techniques.

What Problem Does This Paper Address?

Most digital cameras, especially consumer-grade ones, use single-chip sensors with a Bayer color filter pattern. This pattern captures only one color component (red, green, or blue) for each pixel, resulting in a subsampled image. To compress this video, conventional methods first reconstruct a full RGB image (demosaicing), convert it to YUV format, subsample the chroma components (U and V), and then compress it using standards like H.264/AVC. This process introduces redundancy during demosaicing, leading to higher encoding complexity and potential quality loss. The authors aim to address this by developing a more efficient pre-processing pipeline that minimizes redundancy and computational load while preserving or enhancing video quality.

Methods

The core of the proposed method lies in a modified color space conversion and chroma subsampling strategy. Instead of the standard YUV 4:2:0 subsampling where U and V are sampled at the same locations, the authors propose a YUV 4:2:2 format. Specifically:

  • Luma (Y) Calculation: Y pixels are calculated only at the locations of the green pixels in the Bayer pattern, effectively halving the number of luma pixels to be processed and compressed.
  • Chroma (U and V) Calculation: The positions for calculating U and V are carefully chosen to optimize the reconstruction of red and blue pixels, respectively. V is calculated at the R pixel locations, and U is calculated at the B pixel locations. This is a departure from standard methods where U and V are co-sited.
  • Data Reorganization: The calculated Y, U, and V data, originally in a quincunx pattern, are reorganized into a rectangular format suitable for standard video codecs.
  • Distributed Video Coding (DVC) Integration: To further reduce encoder complexity, the proposed method is integrated with a Wyner-Ziv video coding framework. In DVC, motion estimation is shifted from the encoder to the decoder, drastically lowering the encoder's computational burden.

Datasets

The authors used three custom Bayer-pattern video sequences captured in their laboratory:

  • A sequence with significant panning motion.
  • A sequence with zooming motion.
  • A sequence with an object moving over a static background.

These sequences were used to evaluate the rate-distortion performance of their proposed method against conventional approaches.

Key Insights and Ideas

  • Optimized Chroma Sampling: The key insight is that the standard co-sited chroma sampling in YUV 4:2:0 is not optimal for Bayer-patterned data. By strategically placing U and V samples at locations that are most critical for reconstructing specific color channels (R and B), they improve reconstruction quality.
  • Reduced Luma Processing: Compressing only half the luma pixels significantly cuts down computational complexity without a proportional loss in quality, as the green channel is generally more dominant in perceived luminance.
  • Synergy with DVC: The proposed color space conversion method complements distributed video coding well. While DVC shifts complexity to the decoder, the optimized pre-processing still yields quality improvements, demonstrating the robustness of the approach.
  • Complexity vs. Quality Trade-off: The paper shows that it's possible to achieve both reduced encoding complexity and improved or comparable video quality, challenging the notion that these are mutually exclusive goals in Bayer-pattern video compression.

Key Results

  • Encoding Complexity Reduction: The proposed method reduces the encoder's computational complexity by approximately 50% when used with H.264/AVC. This reduction is even greater when combined with Wyner-Ziv video coding due to the shift of motion estimation to the decoder.
  • Video Quality Improvement: When using H.264/AVC, the proposed scheme achieves a gain of up to 1.5 dB in CPSNR (Composite Peak Signal-to-Noise Ratio) over the conventional method, particularly at higher bit rates. At lower bit rates, the gain is still noticeable, around 0.5 dB.
  • DVC Performance: Similar quality gains are observed when the proposed method is used with a Wyner-Ziv video coder, indicating its effectiveness across different compression paradigms.

Limitations and Future Work

  • Decoder Complexity: While the encoder complexity is reduced, the paper notes that the Wyner-Ziv system, in general, can have higher decoding complexity, although specific implementations might mitigate this.
  • Low Bitrate Performance: At very low bitrates, the proposed method's performance converges to, or slightly underperforms, the conventional method.
  • Future Work: The paper implicitly suggests further exploration of optimal chroma sampling strategies and their integration with advanced DVC techniques for even greater efficiency and quality. The authors also mention that their Wyner-Ziv video codec was developed in-house, implying potential for further optimization.

Summary updated

Abstract

Most consumer digital color cameras capture video using a single chip. Single chip cameras do not capture RGB triples for every pixel, but a subsampled version with only one color component per pixel (e.g. Bayer pattern). Conventionally, a full resolution video is constructed from the Bayer pattern by demosaicing before being converted to YUV domain for compression. In order to lower the encoding complexity, we propose in this work a novel color space conversion in the pre-processing step. Compared to the conventional method, the proposed scheme reduces the encoding complexity almost by half. Moreover, it improves the reconstructed video quality by up to 1.5 dB in CPSNR, when H.264/AVC is used for compression. To further lower the encoding complexity, we additionally use our Wyner-Ziv video coder for compression. Again, we observe in our experiments a similar gain of the proposed scheme over the conventional one.
BibTeX
@inproceedings{chen2008low,
  author = {Hu Chen and Mingzhe Sun and Eckehard Steinbach},
  title = {{Low-complexity Bayer-pattern video compression using distributed video coding}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2008},
  doi = {10.1117/12.807273},
}
Figure