Lightweight Adaptive Feature De-drifting for Compressed Image Classification

Long Peng, Yang Cao, Yuejin Sun, Yang Wang

Abstract

Deep learning models for image classification often suffer significant performance degradation when deployed in real-world scenarios where images are heavily compressed. This paper introduces the Adaptive Feature De-drifting (AFD) Module, a lightweight, "plug-and-play" component designed to mitigate the effects of JPEG compression artifacts on neural network features. By estimating "feature drift" in the frequency domain and employing structural re-parameterization, the AFD-Module restores classification accuracy on mobile devices with minimal computational overhead. Experimental results show that the AFD-Module can improve Top-1 accuracy on ResNet-50 from 31.65% to 49.02% under heavy compression while remaining significantly faster than traditional image restoration methods.

Introduction

While deep learning models achieve state-of-the-art results on high-quality datasets, their performance often collapses when processing images compressed via schemes like JPEG. This degradation is caused by feature drifting: the internal mathematical representations (features) extracted from a compressed image deviate from those extracted from a clean version.

Existing JPEG Artifact Removal (JAR) methods typically fail to solve this for mobile classification for two reasons. First, they are optimized for human vision, often smoothing textures in a way that destroys the fine-grained details necessary for machine classification. Second, their computational complexity makes them unsuitable for real-time pre-processing on resource-constrained mobile hardware. The AFD-Module addresses these gaps by focusing on feature-level restoration rather than pixel-level aesthetics.

The AFD-Module Architecture

The AFD-Module is designed to be integrated into the shallow layers of a pre-trained classifier, where low-level texture features are most vulnerable to compression noise. It consists of two specialized sub-networks:

  1. Feature Drifting Estimation Network (FDE-Net): Rather than operating in the pixel domain, the FDE-Net analyzes images in the Discrete Cosine Transform (DCT) domain. Since JPEG compression occurs in this frequency space, the FDE-Net can more accurately estimate the degree of drift within 8x8 blocks. It produces a Feature Drifting Map (FDM) that serves as a spatial guide for restoration.
  2. Feature Enhancement Network (FE-Net): This lightweight, U-Net-style network takes the degraded features from the backbone model and "de-drifts" them using the guidance of the FDM.

Efficiency via Structural Re-parameterization

To maintain high inference speeds, the authors utilize structural re-parameterization (RepConv). During training, the module uses a complex, multi-branch architecture to learn robust feature representations. For deployment, these branches are mathematically fused into a single, computationally efficient convolution layer. This allows the module to provide the representational power of a large network with the footprint of a much smaller one.

Experimental Evaluation

The AFD-Module was trained on the DIV2K dataset (synthesized into DIV2K-C with various compression levels) and evaluated on benchmarks including ImageNet-C and the AVIF format to test generalization.

Key Findings

  • Accuracy Gains: On a ResNet-50 backbone subjected to heavy JPEG compression (Quality Factor 7), the AFD-Module boosted Top-1 accuracy by over 17 percentage points (31.65% to 49.02%).
  • Mobile Efficiency: The module is approximately 3x faster than leading JAR methods when tested on a Snapdragon 865 mobile CPU. It requires 10 to 100 times fewer FLOPs than traditional models like FBCNN.
  • Adaptive Intervention: Unlike traditional smoothing filters that can hurt accuracy at low compression levels, the AFD-Module is adaptive. It learns to intervene heavily only when significant drift is detected, preserving the integrity of high-quality images.
  • Superiority to Fine-Tuning: The module outperforms standard transfer learning (fine-tuning the entire model on compressed data), which often leads to "catastrophic forgetting" of clean image features.

Conclusion

The AFD-Module represents a significant step forward in making deep learning models robust to real-world data degradation. By leveraging frequency-domain insights and efficient re-parameterization, it provides a practical solution for high-accuracy image classification on mobile devices. Future research may extend this de-drifting logic to other computer vision tasks such as object detection and semantic segmentation in compressed video streams.

Summary updated

Abstract

JPEG is a widely used compression scheme to efficiently reduce the volume of transmitted images. The artifacts appear among blocks due to the information loss, which not only affects the quality of images but also harms the subsequent high-level tasks in terms of feature drifting. High-level vision models trained on high-quality images will suffer performance degradation when dealing with compressed images, especially on mobile devices. Numerous learning-based JPEG artifact removal methods have been proposed to handle visual artifacts. However, it is not an ideal choice to use these JPEG artifact removal methods as a pre-processing for compressed image classification for the following reasons: 1. These methods are designed for human vision rather than high-level vision models; 2. These methods are not efficient enough to serve as pre-processing on resource-constrained devices. To address these issues, this paper proposes a novel lightweight AFD module to boost the performance of pre-trained image classification models when facing compressed images. First, a FDE-Net is devised to generate the spatial-wise FDM in the DCT domain. Next, the estimated FDM is transmitted to the FE-Net to generate the mapping relationship between degraded features and corresponding high-quality features. A simple but effective RepConv block equipped with structural re-parameterization is utilized in FE-Net, which enriches feature representation in the training phase while maintaining efficiency in the deployment phase. After training on limited compressed images, the AFD-Module can serve as a "plug-and-play" model for pre-trained classification models to improve their performance on compressed images. Experiments demonstrate that our proposed AFD module can comprehensively improve the accuracy of the pre-trained classification models and significantly outperform the existing methods.
BibTeX
@article{peng2024lightweight,
  author = {Long Peng and Yang Cao and Yuejin Sun and Yang Wang},
  title = {{Lightweight Adaptive Feature De-drifting for Compressed Image Classification}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2024.3350917},
}
Figure