TL;DR
This paper introduces "Resize-Compress," a novel framework that enhances learned image compression by adaptively estimating the optimal spatial downsampling factor for each image. By jointly training a resizing module with a compression network, they achieve significant improvements in rate-distortion performance, particularly at lower bitrates, without altering the core compression architecture.
What Problem Does This Paper Address?
Traditional image compression techniques often rely on fixed downsampling operations, which can lead to suboptimal rate-distortion (RD) performance. While adaptive resizing has been explored in video coding and some learned compression methods, finding the best reduced resolution in a perceptually meaningful way for learned image compression has been challenging. Existing methods often resort to computationally expensive exhaustive searches over various resolutions. This paper aims to bridge this gap by proposing an efficient, content-adaptive approach to determine the optimal resize factor within an end-to-end learned image compression pipeline.
Methods
The core of the proposed "Resize-Compress" framework involves two key innovations:
- Differentiable Resizing Layers: The authors employ differentiable downsampling and upsampling layers that sandwich the existing learned image compression model. These layers are controlled by a single scalar parameter, $M$, representing the resize factor. This allows gradients to flow through the resizing process during training.
- ResizeParamNet: A lightweight, auxiliary convolutional neural network (CNN) is introduced to predict the optimal resize factor $M$ for each input image. This network is trained jointly with the compression model, enabling content-adaptive resizing without the need for brute-force search. The predicted $M$ is then signaled in the bitstream with minimal overhead.
The framework also incorporates pre- and post-filtering networks to mitigate information loss during downsampling and to enhance reconstruction quality, respectively.
Datasets
- Training: The authors used the CLIC Professional Dataset, consisting of high-quality images. They filtered it to include images with dimensions of at least 384 pixels in both height and width, resulting in a subset of 1600 images. No data augmentation was applied.
- Evaluation:
- Kodak dataset: A standard dataset of 24 high-quality uncompressed images (768x512 pixels).
- Tecnick dataset: A dataset of 100 images (1200x1200 pixels).
- JPEG AI dataset: 16 ultra-high-definition images.
- ImageNet: Used for visualizing learned resizing factors.
None of the test images were included in the training sets to ensure unbiased evaluation.
Key Insights and Ideas
- Content-Adaptive Resizing is Crucial: The paper demonstrates that tailoring the downsampling factor to the specific content of an image (e.g., high-detail areas vs. smooth regions) significantly impacts compression efficiency and perceptual quality.
- Auxiliary Network for Efficiency: Using a dedicated, lightweight network (ResizeParamNet) to predict the resize factor is far more efficient than exhaustive search, making the approach practical for real-time applications.
- Perceptual Quality Matters: While trained with MSE loss, the framework shows substantial gains in perceptual quality metrics (MS-SSIM, VIF, VMAF), suggesting that adaptive resizing effectively targets human perception of image quality.
- Bicubic Interpolation Advantage: Empirically, bicubic interpolation proved more effective for training than bilinear interpolation, likely due to its larger receptive field providing stronger gradient signals.
- Pre- and Post-filters are Beneficial: Both the pre-filter (for downsampling) and post-filter (for upsampling) contribute to improved performance, with the post-filter showing a slightly greater impact.
Key Results
- BD-rate Improvements: The proposed Resize-Compress framework achieved an average Bjøntegaard-Delta rate (BD-rate) improvement of approximately 10% against leading perceptual quality engines when applied to various deep image compression models.
- Perceptual Gains: Significant RD performance gains were observed, especially when measured by perceptual quality metrics like MS-SSIM, VIF, and VMAF, outperforming even more complex compression models in some cases.
- Subjective Study: A subjective user study showed that images compressed with Resize-Compress were preferred by 67.24% of subjects on average, highlighting its perceptual benefits.
- Efficiency: The ResizeParamNet adds minimal computational overhead, making the approach efficient. While encoding/decoding complexity increased slightly, it was deemed negligible for more sophisticated models.
Limitations and Future Work
- PSNR vs. Perceptual Quality: The paper notes that PSNR-based BD-rates did not always show improvements, especially for lower-resolution datasets like Kodak, suggesting PSNR's insensitivity to perceptual quality at high bitrates.
- Failure Cases: The authors acknowledge that there are failure cases where the original model is preferred, particularly when subtle artifacts in the original are less noticeable than those introduced by resizing. This highlights the ongoing challenge of predicting optimal resize factors for diverse image content.
- Computational Overhead: While efficient, the resizing layers do introduce some computational overhead, especially on CPUs. Future work could focus on optimizing these layers using native languages or low-level instructions.
- Generalizability to Non-Differentiable Codecs: The authors suggest that the learned ResizeParamNet could potentially be adapted as a "fast estimator" for optimal resize factors in non-differentiable hybrid video codecs.