Abstract
Deep Convolutional Neural Networks (DCNNs) are highly effective on high-quality datasets but exhibit significant fragility when faced with real-world image corruptions such as blur, noise, and adverse weather. This vulnerability stems from a distribution shift where features from low-quality data diverge from those learned during clean training. We introduce a novel "plug-and-play" module that integrates Vector Quantization (VQ) and Self-Attention (SA) to map features into a shared discrete codebook. This mechanism forces the model to learn quality-independent representations by focusing on underlying object prototypes rather than corruption-induced artifacts. Our method achieves a state-of-the-art 43.1% mean Corruption Error (mCE) on the ImageNet-C benchmark while maintaining computational efficiency.
Introduction
Standard DCNNs, such as ResNet50, can experience accuracy drops from 76% to below 30% when images are contaminated with Gaussian noise. Previous attempts to mitigate this—ranging from data augmentation to feature distillation—often fail to explicitly map low-quality features back to their high-quality counterparts. The core challenge lies in the continuous nature of feature spaces, where noise can easily shift representations away from the intended class manifold.
Methodology
The proposed framework enhances standard feature extraction through a two-stage process:
- Vector Quantization (VQ): We introduce a learnable "codebook" $E \in R^{n \times d}$ containing feature prototypes. For any extracted feature $z$, the VQ module identifies the nearest prototype $\hat{z}$ in the codebook. By training both clean and corrupted images to map to the same discrete entries, the network learns to treat discretization as a natural denoising filter.
- Self-Attention (SA) Fusion: To prevent the loss of fine-grained detail inherent in pure quantization, the original feature $z$ is concatenated with the quantized prototype $\hat{z}$. A Self-Attention module then adaptively fuses these representations, allowing the model to prioritize stable, prototype-based information while retaining necessary high-level details.
The system is optimized using a composite loss function including Cross-Entropy for classification, VQ loss for codebook updates, and a Commitment loss to ensure backbone features remain aligned with the codebook space.
Experimental Results
The method was evaluated on several rigorous benchmarks, including ImageNet-C (corruption robustness), ImageNet-R (generalization to artistic renditions), and ImageNet-A (natural adversarial images).
Robustness and Generalization
On ImageNet-C, the ResNet50-based model achieved a 43.1% mCE, outperforming previous state-of-the-art methods like QualNet. Furthermore, the model showed a 2% accuracy improvement on ImageNet-R, suggesting that quality-independent features also generalize better across different visual domains (e.g., sketches, paintings).
Salience Preservation
Grad-CAM visualizations demonstrate that the VQ-SA module effectively anchors the model's attention. While vanilla models often shift their focus toward background noise or blurred regions in corrupted images, our model consistently maintains focus on the primary object.
Conclusion
By combining the denoising properties of discrete vector quantization with the adaptive fusion of self-attention, we provide a robust solution for quality-independent representation learning. The module is lightweight and easily integrated into existing architectures. Future work will explore the interaction of these modules with Vision Transformer (ViT) architectures and further optimization of the codebook structure.
Summary updated