Edge-Aware Extended Star-Tetrix Transforms for CFA-Sampled Raw Camera Image Compression

Taizo Suzuki, Liping HuangarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces Extended Star-Tetrix Transforms (XSTTs) and their edge-aware variants (EXSTTs) to improve the compression of raw camera data. By adapting to image edges without requiring extra metadata (side information), these transforms achieve significantly better decorrelation and higher compression efficiency than the standard Star-Tetrix transform used in JPEG XS.

What Problem Does This Paper Address?

Most digital cameras capture images using a single sensor covered by a Color Filter Array (CFA), the most common being the Bayer pattern. This means each pixel only records one color (Red, Green, or Blue). Traditionally, cameras "demosaic" this data immediately—interpolating the missing colors to create a full RGB image—before compressing it. However, demosaicing triples the data volume and introduces redundant information.

A more efficient approach is "compression-first," where the raw CFA data is compressed directly. To do this effectively, researchers use spectral-spatial transforms to convert RGB data into a decorrelated space (like YCbCr), where the components are easier to compress. The challenge is that existing transforms, such as the Star-Tetrix Transform (STT) used in the JPEG XS standard, often ignore local image features like edges. When a transform "predicts" a pixel value across a sharp edge, the prediction error increases, leading to larger file sizes.

Methods

The authors propose two major advancements:

  1. Extended Star-Tetrix Transforms (XSTTs): They generalize the original STT into two new versions. Type-I is a more sophisticated version of previous wavelet-based transforms, while Type-II is a simplified, lower-complexity version. Both utilize 5/3 and 9/7 wavelet filters.
  2. Edge-Aware Weighting (EXSTTs): This is the core innovation. The authors break down the 2D "predict" steps of the wavelet transform into combinations of 1D diagonal or horizontal-vertical transforms. They then apply weights to these transforms based on the direction of edges in the image.
  3. Zero Side Information: Crucially, these edge weights are not stored in the file. Instead, the decoder recalculates the weights on the fly using previously decoded neighboring pixels. This allows for edge-adaptive compression without the "bit tax" of extra metadata.

Datasets

The researchers evaluated their methods using 30 test images:

  • High-Quality Images: 20 images from the MIT-Adobe FiveK dataset (12-bit or 14-bit depth), split into ten "general" images and ten "edge-heavy" images.
  • Mobile Images: 10 images captured with an iPhone SE (3rd generation) to test performance on noisier, small-sensor data.
  • Kodak Dataset: Used for initial pattern simulations and visual examples.

Key Insights and Ideas

  • The "Free" Edge Adaptation: The most significant insight is that edge-aware filtering can be implemented causally. Because the decoder has access to the same local neighborhood as the encoder, it can replicate the edge-detection logic perfectly, making the transform "adaptive" without needing side information.
  • Energy Reduction in $D_g$: The "Difference Green" ($D_g$) component essentially represents the error in the transform's prediction. By weighting the transform along edges, the authors reduced the energy (and thus the bit cost) of this component by up to 30% in high-quality images.
  • Type-I vs. Type-II: While Type-I is more complex and generally performs better on high-quality data, Type-II offers a "sweet spot" for applications requiring lower computational overhead while still outperforming the standard STT.

Key Results

  • Lossless Compression: For images with many edges, the Type-I EXSTT improved the average bitrate by 0.03–0.19 bits per pixel (bpp) compared to conventional methods.
  • Lossy Compression: The XSTTs improved the Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR)—a measure of compression efficiency—by 0.16–0.96 dB for high-quality images.
  • Decorrelation Efficiency: The EXSTTs reduced the energy difference between green components by 3.38–30.08% for high-quality images and 8.97–14.47% for mobile images.
  • Visual Quality: Qualitative results showed no noticeable artifacts even at relatively high compression ratios (e.g., 2 bpp).

Limitations and Future Work

  • Lossy Compression Mismatch: In lossy mode, the weights calculated at the decoder might slightly differ from the encoder due to quantization errors (noise introduced by compression). This can degrade performance at very low bitrates.
  • Noise Sensitivity: On mobile phone images, which are naturally noisier, the edge-aware weights were less effective than on high-quality DSLR data. The authors' previous work still performed slightly better in high-noise scenarios.
  • Simplifications: For this study, the authors omitted white-balancing constants and gamma correction within the transform itself to focus on the mathematical extension of the STT. Integrating these could be a future step for production-ready codecs.

Summary updated

Abstract

Codecs using spectral-spatial transforms efficiently compress raw camera images captured with a color filter array (CFA-sampled raw images) by changing their RGB color space into a decorrelated color space. This study describes two types of spectral-spatial transform, called extended Star-Tetrix transforms (XSTTs), and their edge-aware versions, called edge-aware XSTTs (EXSTTs), with no extra bits (side information) and little extra complexity. They are obtained by (i) extending the Star-Tetrix transform (STT), which is one of the latest spectral-spatial transforms, to a new version of our previously proposed wavelet-based spectral-spatial transform and a simpler version, (ii) considering that each 2-D predict step of the wavelet transform is a combination of two 1-D diagonal or horizontal-vertical transforms, and (iii) weighting the transforms along the edge directions in the images. Compared with XSTTs, the EXSTTs can decorrelate CFA-sampled raw images well: they reduce the difference in energy between the two green components by about $3.38$--$30.08$ \% for high-quality camera images and $8.97$--$14.47$ \% for mobile phone images. The experiments on JPEG 2000-based lossless and lossy compression of CFA-sampled raw images show better performance than conventional methods. For high-quality camera images, the XSTTs/EXSTTs produce results equal to or better than the conventional methods: especially for images with many edges, the type-I EXSTT improves them by about $0.03$--$0.19$ bpp in average lossless bitrate and the XSTTs improve them by about $0.16$--$0.96$ dB in average Bjøntegaard delta peak signal-to-noise ratio. For mobile phone images, our previous work perform the best, whereas the XSTTs/EXSTTs show similar trends to the case of high-quality camera images.
BibTeX
@misc{suzuki2022edge,
  author = {Taizo Suzuki and Liping Huang},
  title = {{Edge-Aware Extended Star-Tetrix Transforms for CFA-Sampled Raw Camera Image Compression}},
  year = {2022},
  eprint = {2209.00932},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.00932},
}
Figure