TL;DR
This paper introduces Extended Star-Tetrix Transforms (XSTTs) and their edge-aware variants (EXSTTs) to improve the compression of raw camera data. By adapting to image edges without requiring extra metadata (side information), these transforms achieve significantly better decorrelation and higher compression efficiency than the standard Star-Tetrix transform used in JPEG XS.
What Problem Does This Paper Address?
Most digital cameras capture images using a single sensor covered by a Color Filter Array (CFA), the most common being the Bayer pattern. This means each pixel only records one color (Red, Green, or Blue). Traditionally, cameras "demosaic" this data immediately—interpolating the missing colors to create a full RGB image—before compressing it. However, demosaicing triples the data volume and introduces redundant information.
A more efficient approach is "compression-first," where the raw CFA data is compressed directly. To do this effectively, researchers use spectral-spatial transforms to convert RGB data into a decorrelated space (like YCbCr), where the components are easier to compress. The challenge is that existing transforms, such as the Star-Tetrix Transform (STT) used in the JPEG XS standard, often ignore local image features like edges. When a transform "predicts" a pixel value across a sharp edge, the prediction error increases, leading to larger file sizes.
Methods
The authors propose two major advancements:
- Extended Star-Tetrix Transforms (XSTTs): They generalize the original STT into two new versions. Type-I is a more sophisticated version of previous wavelet-based transforms, while Type-II is a simplified, lower-complexity version. Both utilize 5/3 and 9/7 wavelet filters.
- Edge-Aware Weighting (EXSTTs): This is the core innovation. The authors break down the 2D "predict" steps of the wavelet transform into combinations of 1D diagonal or horizontal-vertical transforms. They then apply weights to these transforms based on the direction of edges in the image.
- Zero Side Information: Crucially, these edge weights are not stored in the file. Instead, the decoder recalculates the weights on the fly using previously decoded neighboring pixels. This allows for edge-adaptive compression without the "bit tax" of extra metadata.
Datasets
The researchers evaluated their methods using 30 test images:
- High-Quality Images: 20 images from the MIT-Adobe FiveK dataset (12-bit or 14-bit depth), split into ten "general" images and ten "edge-heavy" images.
- Mobile Images: 10 images captured with an iPhone SE (3rd generation) to test performance on noisier, small-sensor data.
- Kodak Dataset: Used for initial pattern simulations and visual examples.
Key Insights and Ideas
- The "Free" Edge Adaptation: The most significant insight is that edge-aware filtering can be implemented causally. Because the decoder has access to the same local neighborhood as the encoder, it can replicate the edge-detection logic perfectly, making the transform "adaptive" without needing side information.
- Energy Reduction in $D_g$: The "Difference Green" ($D_g$) component essentially represents the error in the transform's prediction. By weighting the transform along edges, the authors reduced the energy (and thus the bit cost) of this component by up to 30% in high-quality images.
- Type-I vs. Type-II: While Type-I is more complex and generally performs better on high-quality data, Type-II offers a "sweet spot" for applications requiring lower computational overhead while still outperforming the standard STT.
Key Results
- Lossless Compression: For images with many edges, the Type-I EXSTT improved the average bitrate by 0.03–0.19 bits per pixel (bpp) compared to conventional methods.
- Lossy Compression: The XSTTs improved the Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR)—a measure of compression efficiency—by 0.16–0.96 dB for high-quality images.
- Decorrelation Efficiency: The EXSTTs reduced the energy difference between green components by 3.38–30.08% for high-quality images and 8.97–14.47% for mobile images.
- Visual Quality: Qualitative results showed no noticeable artifacts even at relatively high compression ratios (e.g., 2 bpp).
Limitations and Future Work
- Lossy Compression Mismatch: In lossy mode, the weights calculated at the decoder might slightly differ from the encoder due to quantization errors (noise introduced by compression). This can degrade performance at very low bitrates.
- Noise Sensitivity: On mobile phone images, which are naturally noisier, the edge-aware weights were less effective than on high-quality DSLR data. The authors' previous work still performed slightly better in high-noise scenarios.
- Simplifications: For this study, the authors omitted white-balancing constants and gamma correction within the transform itself to focus on the mathematical extension of the STT. Integrating these could be a future step for production-ready codecs.
Summary updated