Abstract: Images are typically transmitted or stored in compressed formats, with most AI tasks performed after reconstruction. While CNN-based image compression is surpassing heuristic methods like JPEG, the computational complexity of reconstruction remains a significant bottleneck. This paper demonstrates that learning directly from the compressed domain can achieve comparable or superior accuracy to traditional reconstructed-domain learning. At a high compression rate of 0.098 bpp, the proposed system provides a 3% absolute accuracy boost over traditional flows. This improvement is driven by optimizing for original-sized images rather than standardized crops, alongside an efficient model-free entropy estimation method and a feature selection criterion that reduces transmission and computation costs.
1. Introduction: The "Reconstruction Tax"
Modern image compression is shifting from heuristic methods like JPEG to neural-network-based approaches. While these CNN-based codecs offer superior bit-rates, they impose a massive computational burden. For example, a state-of-the-art reconstruction network might require over 900 GFLOPS to process a 1080p image—dwarfing the 4 GFLOPS needed for a standard ResNet-50 inference.
Furthermore, most computer vision models are trained on fixed-size inputs (e.g., 224x224). In real-world scenarios, images vary in resolution. Traditionally, images are decompressed and resized before inference, a process that is slow and discards valuable information. This research aims to bridge this gap by learning directly from the "latent" features of the compressed domain.
2. Methodology
The authors propose a multi-task system consisting of three main components: a CompNet (compression), a RecNet (reconstruction, used primarily during training), and a LearnNet (task-specific inference).
2.1 Key Innovations
- Model-Free Entropy Estimation: To optimize for bit-rate, the authors replaced complex entropy-estimation networks with a differentiable proxy: the differential entropy of a Gaussian distribution ($\log \sigma^2$). This variance-based metric shows a 99% correlation with true discrete entropy, significantly streamlining training.
- Variable-Size Training: Instead of resizing compressed features to a fixed square, the models were trained using a "variable-size" strategy. By randomly resizing and cropping training images to various dimensions (between 224 and 448), the network becomes robust to real-world aspect ratios and resolutions.
- Feature Selection: The authors demonstrated that not all 256 compressed feature maps are necessary for vision tasks. Using variance as a criterion for "information richness," they achieved similar accuracy using only 1/8th (32) of the original feature maps, further reducing computation.
3. Experimental Results
The framework was validated using ImageNet for classification, PASCAL VOC 2012 for segmentation, and the Kodak dataset for entropy validation.
- Superior Accuracy: At a high compression rate of 0.098 bits per pixel (bpp), the system achieved a 3.29% absolute accuracy boost on ImageNet compared to the traditional "reconstruct-then-classify" flow.
- Efficiency and Robustness: By skipping reconstruction, the system avoids hundreds of GFLOPS of latency. Additionally, variable-size training improved top-1 accuracy by roughly 4% over models trained on fixed-size inputs when tested on original-sized images.
- Competitive Bit-rates: The system outperformed traditional JPEG and JPEG2000 in terms of the trade-off between image quality (PSNR) and bit-rate.
4. Conclusion and Future Work
This research demonstrates that reconstruction is often an unnecessary middleman for machine vision. By learning from compressed features, models access raw information before reconstruction artifacts are introduced. While the current system uses a fixed downsampling factor (1/16), future work could explore adaptive scaling for ultra-high-resolution (4K) datasets, where the benefits of compressed-domain learning are expected to be even more pronounced.
Summary updated