SLIC: Self-Conditioned Adaptive Transform with Large-Scale Receptive Fields for Learned Image Compression

Wei Jiang, Peirong Ning, Ronggang WangarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces LLIC, a novel learned image compression framework that significantly enhances performance by employing large receptive fields within convolutional neural networks (CNNs) and making these kernels adapt to the input data. LLIC achieves state-of-the-art results, outperforming existing methods by reducing bitrate while maintaining or improving image quality, especially for high-resolution images.

What Problem Does This Paper Address?

Traditional image compression methods rely on fixed transforms, limiting their ability to adapt to diverse image content. Learned image compression (LIC) models, often based on variational autoencoders (VAEs), have shown promise by using neural networks for transforms and entropy models. However, existing LIC models typically use small convolutional kernels, resulting in limited effective receptive fields (ERFs). While transformers offer larger receptive fields and dynamic weights, their quadratic complexity hinders their use in high-resolution image coding. This paper aims to bridge this gap by enabling CNNs to achieve large receptive fields and content adaptability, crucial for efficient compression, particularly for high-resolution images, without incurring prohibitive computational costs.

Methods

The core of LLIC lies in its novel Spatial Transform Block (STB) and Channel Transform Block (CTB).

  • Large Receptive Fields: LLIC introduces large-kernel depth-wise convolutions (up to 11x11) within the STB. This is a departure from typical LIC methods that use smaller kernels or rely on attention mechanisms. Depth-wise convolutions help manage the computational complexity associated with large kernels.
  • Self-Conditioned Adaptability: To make the large kernels adaptive, LLIC generates their weights dynamically based on the input features. This "self-conditioned" approach allows the network to tailor its transform to the specific content of the image, improving redundancy removal. This is achieved through a "Condition Branch" that processes the input features to produce the kernel weights.
  • Non-linear Embedding and Gate Mechanism: The STB also incorporates a Depth-wise Residual Block (DepthRB) for non-linear embedding and a Gate Block for efficient point-wise interactions, enhancing the model's expressiveness and efficiency.
  • Adaptive Channel-wise Bit Allocation: The CTB addresses channel-wise redundancy by dynamically allocating bits based on channel importance. It uses a similar self-conditioned mechanism to generate channel importance factors, allowing more bits to be allocated to informative channels.
  • Improved Training: To fully leverage the large kernels, LLIC employs a two-stage training strategy using larger image patches (512x512) after an initial training phase with smaller patches (256x256).

LLIC integrates these novel blocks with existing entropy models (STF, ELIC, LIC-TCM) to create LLIC-STF, LLIC-ELIC, and LLIC-TCM, respectively, for fair comparison.

Datasets

The models were trained on a combination of large-scale datasets:

  • COCO2017
  • ImageNet
  • DIV2K
  • Flicker2K

For evaluation, the models were tested on several standard datasets, including:

  • Kodak [59]
  • Tecnick [64]
  • CLIC Pro Valid [19]
  • CLIC 2021 Test [66]
  • JPEGAI Test [67]

These datasets vary in resolution, from 768x512 to 3680x2456, allowing for a comprehensive evaluation of the models' performance on different image sizes.

Key Insights and Ideas

  • Large Kernels in CNNs for Compression: The paper demonstrates that large receptive fields, traditionally a strength of transformers, can be effectively integrated into CNN-based LIC models using depth-wise convolutions. This offers a path to achieving transformer-like performance with CNN efficiency.
  • Self-Conditioned Weights are Crucial: Making convolutional weights adaptive to the input data (self-conditioned) significantly improves performance. This adaptability is key to handling diverse image content and reducing redundancy more effectively than static kernels.
  • Channel-wise Adaptability Matters: Beyond spatial redundancy, addressing channel-wise redundancy through adaptive bit allocation (as done in the CTB) further boosts compression efficiency.
  • Large Patches for Large Kernels: Training with larger image patches is essential to fully exploit the benefits of large convolutional kernels, as smaller patches do not provide sufficient context.

Key Results

  • State-of-the-Art Performance: LLIC models (LLIC-STF, LLIC-ELIC, LLIC-TCM) achieve significant improvements over their baselines and other state-of-the-art methods.
  • BD-Rate Reduction: LLIC models reduce the BD-Rate by an average of 9.49% (LLIC-STF), 9.47% (LLIC-ELIC), and 10.94% (LLIC-TCM) on the Kodak dataset compared to VTM-17.0 Intra.
  • Improved ERF: Visualizations show that LLIC models achieve significantly larger Effective Receptive Fields (ERFs) compared to their baselines, indicating better capture of spatial context.
  • Computational Efficiency: Despite using large kernels, LLIC maintains modest complexity, especially compared to transformer-based methods. For instance, LLIC-TCM achieves comparable or better performance than LIC-TCM Large with roughly half the MACs.
  • Faster Encoding/Decoding: LLIC models exhibit fast encoding and decoding speeds, making them practical for real-world applications.

Limitations and Future Work

  • Complexity: While LLIC is more efficient than transformers, its complexity is still not low enough for direct deployment on mobile devices.
  • Generalization: Models trained on natural images might have limited generalization to out-of-distribution data like screen content.
  • Future Work: The authors suggest integrating more advanced entropy models and exploring asymmetric encoder-decoder structures for further complexity reduction. They also propose fine-tuning the encoder or latent representation for specific input types to improve performance on natural images and potentially enhance generalization.

Summary updated

Abstract

Recently, transformers are trending as replacements for CNNs in vision tasks, including compression. This trend compels us to question the inherent limitations of CNNs compared to transformers and to explore if CNNs can be enhanced to achieve the same or even better performance than transformers. We want to design a pure CNN based model for compression as most devices are optimized for CNNs well. In our analysis, we find that the key strengths of transformers lie in their dynamic weights and large receptive fields. To enable CNNs with such properties, we propose a novel transform module with large receptive filed learning and self-conditioned adaptability for learned image compression, named SLIC. Specifically, we enlarge the receptive field of depth-wise convolution with suitable complexity and generate the weights according to given conditions. In addition, we also investigate the self-conditioned factor for channels. To prove the effectiveness of our proposed transform module, we equip it with existing entropy models ChARM, SCCTX, and SWAtten and we obtain models SLIC-ChARM, SLIC-SCCTX, and SLIC-SWAtten. Extensive experiments demonstrate our SLIC-ChARM, SLIC-SCCTX, and SLIC-SWAtten have significant improvements over corresponding baselines and achieve SOTA performances with suitable complexity on 5 test datasets (Kodak, Tecnick, CLIC 20, CLIC 21, JPEGAI). Code will be available at https://github.com/JiangWeibeta/SLIC.
BibTeX
@misc{jiang2023slic,
  author = {Wei Jiang and Peirong Ning and Ronggang Wang},
  title = {{SLIC: Self-Conditioned Adaptive Transform with Large-Scale Receptive Fields for Learned Image Compression}},
  year = {2023},
  eprint = {2304.09571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.09571},
}
Figure