Lightweight Adaptive Feature De-Drifting for Compressed Image Classification

Long Peng, Yang Cao, Yuejin Sun, Yang Wang

Abstract

Deep learning models for image classification often suffer significant performance degradation when deployed in real-world scenarios where images are heavily compressed. This paper introduces the Adaptive Feature De-drifting (AFD) Module, a lightweight, "plug-and-play" component designed to mitigate the effects of JPEG compression artifacts on neural network features. By estimating "feature drift" in the frequency domain and employing structural re-parameterization, the AFD-Module restores classification accuracy on mobile devices with minimal computational overhead. Experimental results show that the AFD-Module can improve Top-1 accuracy on ResNet-50 from 31.65% to 49.02% under heavy compression while remaining significantly faster than traditional image restoration methods.

Introduction

While deep learning models achieve state-of-the-art results on high-quality datasets, their performance often collapses when processing images compressed via schemes like JPEG. This degradation is caused by feature drifting: the internal mathematical representations (features) extracted from a compressed image deviate from those extracted from a clean version.

Existing JPEG Artifact Removal (JAR) methods typically fail to solve this for mobile classification for two reasons. First, they are optimized for human vision, often smoothing textures in a way that destroys the fine-grained details necessary for machine classification. Second, their computational complexity makes them unsuitable for real-time pre-processing on resource-constrained mobile hardware. The AFD-Module addresses these gaps by focusing on feature-level restoration rather than pixel-level aesthetics.

The AFD-Module Architecture

The AFD-Module is designed to be integrated into the shallow layers of a pre-trained classifier, where low-level texture features are most vulnerable to compression noise. It consists of two specialized sub-networks:

  1. Feature Drifting Estimation Network (FDE-Net): Rather than operating in the pixel domain, the FDE-Net analyzes images in the Discrete Cosine Transform (DCT) domain. Since JPEG compression occurs in this frequency space, the FDE-Net can more accurately estimate the degree of drift within 8x8 blocks. It produces a Feature Drifting Map (FDM) that serves as a spatial guide for restoration.
  2. Feature Enhancement Network (FE-Net): This lightweight, U-Net-style network takes the degraded features from the backbone model and "de-drifts" them using the guidance of the FDM.

Efficiency via Structural Re-parameterization

To maintain high inference speeds, the authors utilize structural re-parameterization (RepConv). During training, the module uses a complex, multi-branch architecture to learn robust feature representations. For deployment, these branches are mathematically fused into a single, computationally efficient convolution layer. This allows the module to provide the representational power of a large network with the footprint of a much smaller one.

Experimental Evaluation

The AFD-Module was trained on the DIV2K dataset (synthesized into DIV2K-C with various compression levels) and evaluated on benchmarks including ImageNet-C and the AVIF format to test generalization.

Key Findings

  • Accuracy Gains: On a ResNet-50 backbone subjected to heavy JPEG compression (Quality Factor 7), the AFD-Module boosted Top-1 accuracy by over 17 percentage points (31.65% to 49.02%).
  • Mobile Efficiency: The module is approximately 3x faster than leading JAR methods when tested on a Snapdragon 865 mobile CPU. It requires 10 to 100 times fewer FLOPs than traditional models like FBCNN.
  • Adaptive Intervention: Unlike traditional smoothing filters that can hurt accuracy at low compression levels, the AFD-Module is adaptive. It learns to intervene heavily only when significant drift is detected, preserving the integrity of high-quality images.
  • Superiority to Fine-Tuning: The module outperforms standard transfer learning (fine-tuning the entire model on compressed data), which often leads to "catastrophic forgetting" of clean image features.

Conclusion

The AFD-Module represents a significant step forward in making deep learning models robust to real-world data degradation. By leveraging frequency-domain insights and efficient re-parameterization, it provides a practical solution for high-accuracy image classification on mobile devices. Future research may extend this de-drifting logic to other computer vision tasks such as object detection and semantic segmentation in compressed video streams.

Summary updated

BibTeX
@article{peng2024lightweight,
  author = {Long Peng and Yang Cao and Yuejin Sun and Yang Wang},
  title = {{Lightweight Adaptive Feature De-Drifting for Compressed Image Classification}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2024.3350917},
}
Figure