Learning from the CNN-based Compressed Domain

Zhenzhen Wang, Minghai Qin, Yen-Kuang ChenWACV 2022 · OpenReview

Abstract: Images are typically transmitted or stored in compressed formats, with most AI tasks performed after reconstruction. While CNN-based image compression is surpassing heuristic methods like JPEG, the computational complexity of reconstruction remains a significant bottleneck. This paper demonstrates that learning directly from the compressed domain can achieve comparable or superior accuracy to traditional reconstructed-domain learning. At a high compression rate of 0.098 bpp, the proposed system provides a 3% absolute accuracy boost over traditional flows. This improvement is driven by optimizing for original-sized images rather than standardized crops, alongside an efficient model-free entropy estimation method and a feature selection criterion that reduces transmission and computation costs.

1. Introduction: The "Reconstruction Tax"

Modern image compression is shifting from heuristic methods like JPEG to neural-network-based approaches. While these CNN-based codecs offer superior bit-rates, they impose a massive computational burden. For example, a state-of-the-art reconstruction network might require over 900 GFLOPS to process a 1080p image—dwarfing the 4 GFLOPS needed for a standard ResNet-50 inference.

Furthermore, most computer vision models are trained on fixed-size inputs (e.g., 224x224). In real-world scenarios, images vary in resolution. Traditionally, images are decompressed and resized before inference, a process that is slow and discards valuable information. This research aims to bridge this gap by learning directly from the "latent" features of the compressed domain.

2. Methodology

The authors propose a multi-task system consisting of three main components: a CompNet (compression), a RecNet (reconstruction, used primarily during training), and a LearnNet (task-specific inference).

Comparison of learning workflows: (a) traditional reconstruction-based learning vs. (b) the proposed compressed-domain learning framework.

2.1 Key Innovations

  • Model-Free Entropy Estimation: To optimize for bit-rate, the authors replaced complex entropy-estimation networks with a differentiable proxy: the differential entropy of a Gaussian distribution ($\log \sigma^2$). This variance-based metric shows a 99% correlation with true discrete entropy, significantly streamlining training.
  • Variable-Size Training: Instead of resizing compressed features to a fixed square, the models were trained using a "variable-size" strategy. By randomly resizing and cropping training images to various dimensions (between 224 and 448), the network becomes robust to real-world aspect ratios and resolutions.
  • Feature Selection: The authors demonstrated that not all 256 compressed feature maps are necessary for vision tasks. Using variance as a criterion for "information richness," they achieved similar accuracy using only 1/8th (32) of the original feature maps, further reducing computation.

3. Experimental Results

The framework was validated using ImageNet for classification, PASCAL VOC 2012 for segmentation, and the Kodak dataset for entropy validation.

Rate-distortion performance (PSNR vs. bpp) showing the proposed compression method outperforming JPEG and JPEG2000.
  • Superior Accuracy: At a high compression rate of 0.098 bits per pixel (bpp), the system achieved a 3.29% absolute accuracy boost on ImageNet compared to the traditional "reconstruct-then-classify" flow.
  • Efficiency and Robustness: By skipping reconstruction, the system avoids hundreds of GFLOPS of latency. Additionally, variable-size training improved top-1 accuracy by roughly 4% over models trained on fixed-size inputs when tested on original-sized images.
  • Competitive Bit-rates: The system outperformed traditional JPEG and JPEG2000 in terms of the trade-off between image quality (PSNR) and bit-rate.

4. Conclusion and Future Work

This research demonstrates that reconstruction is often an unnecessary middleman for machine vision. By learning from compressed features, models access raw information before reconstruction artifacts are introduced. While the current system uses a fixed downsampling factor (1/16), future work could explore adaptive scaling for ultra-high-resolution (4K) datasets, where the benefits of compressed-domain learning are expected to be even more pronounced.

Summary updated

Abstract

Images are transmitted or stored in their compressed form and most of the AI tasks are performed from the re-constructed domain. Convolutional neural network (CNN)-based image compression and reconstruction is growing rapidly and it achieves or surpasses the state-of-the-art heuristic image compression methods, such as JPEG or BPG. A major limitation of the application of the CNN-based image compression is on the computation complexity during compression and reconstruction. Therefore, learning from the compressed domain is desirable to avoid the computation and latency caused by reconstruction. In this paper, we show that learning from the compressed domain can achieve comparative or even better accuracy than from the reconstructed domain. At a high compression rate of 0.098 bpp, for example, the proposed compression-learning system has over 3% absolute accuracy boost over the traditional compression-reconstruction-learning flow. The improvement is achieved by optimizing the compression-learning system targeting original-sized instead of standardized (e.g., 224x224) images, which is crucial in practice since real-world images into the system have different sizes. We also propose an efficient model-free entropy estimation method and a criterion to learn from a selected subset of features in the compressed domain to further re-duce the transmission and computation cost without accuracy degradation.
BibTeX
@inproceedings{wang2022learning,
  author = {Zhenzhen Wang and Minghai Qin and Yen-Kuang Chen},
  title = {{Learning from the CNN-based Compressed Domain}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=UwVFCfI22j},
}
Figure