Abstract
Deep learning models for image classification often suffer significant performance degradation when deployed in real-world scenarios where images are heavily compressed. This paper introduces the Adaptive Feature De-drifting (AFD) Module, a lightweight, "plug-and-play" component designed to mitigate the effects of JPEG compression artifacts on neural network features. By estimating "feature drift" in the frequency domain and employing structural re-parameterization, the AFD-Module restores classification accuracy on mobile devices with minimal computational overhead. Experimental results show that the AFD-Module can improve Top-1 accuracy on ResNet-50 from 31.65% to 49.02% under heavy compression while remaining significantly faster than traditional image restoration methods.
Introduction
While deep learning models achieve state-of-the-art results on high-quality datasets, their performance often collapses when processing images compressed via schemes like JPEG. This degradation is caused by feature drifting: the internal mathematical representations (features) extracted from a compressed image deviate from those extracted from a clean version.
Existing JPEG Artifact Removal (JAR) methods typically fail to solve this for mobile classification for two reasons. First, they are optimized for human vision, often smoothing textures in a way that destroys the fine-grained details necessary for machine classification. Second, their computational complexity makes them unsuitable for real-time pre-processing on resource-constrained mobile hardware. The AFD-Module addresses these gaps by focusing on feature-level restoration rather than pixel-level aesthetics.
The AFD-Module Architecture
The AFD-Module is designed to be integrated into the shallow layers of a pre-trained classifier, where low-level texture features are most vulnerable to compression noise. It consists of two specialized sub-networks:
- Feature Drifting Estimation Network (FDE-Net): Rather than operating in the pixel domain, the FDE-Net analyzes images in the Discrete Cosine Transform (DCT) domain. Since JPEG compression occurs in this frequency space, the FDE-Net can more accurately estimate the degree of drift within 8x8 blocks. It produces a Feature Drifting Map (FDM) that serves as a spatial guide for restoration.
- Feature Enhancement Network (FE-Net): This lightweight, U-Net-style network takes the degraded features from the backbone model and "de-drifts" them using the guidance of the FDM.
Efficiency via Structural Re-parameterization
To maintain high inference speeds, the authors utilize structural re-parameterization (RepConv). During training, the module uses a complex, multi-branch architecture to learn robust feature representations. For deployment, these branches are mathematically fused into a single, computationally efficient convolution layer. This allows the module to provide the representational power of a large network with the footprint of a much smaller one.
Experimental Evaluation
The AFD-Module was trained on the DIV2K dataset (synthesized into DIV2K-C with various compression levels) and evaluated on benchmarks including ImageNet-C and the AVIF format to test generalization.
Key Findings
- Accuracy Gains: On a ResNet-50 backbone subjected to heavy JPEG compression (Quality Factor 7), the AFD-Module boosted Top-1 accuracy by over 17 percentage points (31.65% to 49.02%).
- Mobile Efficiency: The module is approximately 3x faster than leading JAR methods when tested on a Snapdragon 865 mobile CPU. It requires 10 to 100 times fewer FLOPs than traditional models like FBCNN.
- Adaptive Intervention: Unlike traditional smoothing filters that can hurt accuracy at low compression levels, the AFD-Module is adaptive. It learns to intervene heavily only when significant drift is detected, preserving the integrity of high-quality images.
- Superiority to Fine-Tuning: The module outperforms standard transfer learning (fine-tuning the entire model on compressed data), which often leads to "catastrophic forgetting" of clean image features.
Conclusion
The AFD-Module represents a significant step forward in making deep learning models robust to real-world data degradation. By leveraging frequency-domain insights and efficient re-parameterization, it provides a practical solution for high-accuracy image classification on mobile devices. Future research may extend this de-drifting logic to other computer vision tasks such as object detection and semantic segmentation in compressed video streams.
Summary updated