Vector Quantization with Self-Attention for Quality-Independent Representation Learning

Zhou Yang, Weisheng Dong, Xin Li, Mengluan Huang, Yulin Sun, Guangming Shi

Abstract

Deep Convolutional Neural Networks (DCNNs) are highly effective on high-quality datasets but exhibit significant fragility when faced with real-world image corruptions such as blur, noise, and adverse weather. This vulnerability stems from a distribution shift where features from low-quality data diverge from those learned during clean training. We introduce a novel "plug-and-play" module that integrates Vector Quantization (VQ) and Self-Attention (SA) to map features into a shared discrete codebook. This mechanism forces the model to learn quality-independent representations by focusing on underlying object prototypes rather than corruption-induced artifacts. Our method achieves a state-of-the-art 43.1% mean Corruption Error (mCE) on the ImageNet-C benchmark while maintaining computational efficiency.

Introduction

Standard DCNNs, such as ResNet50, can experience accuracy drops from 76% to below 30% when images are contaminated with Gaussian noise. Previous attempts to mitigate this—ranging from data augmentation to feature distillation—often fail to explicitly map low-quality features back to their high-quality counterparts. The core challenge lies in the continuous nature of feature spaces, where noise can easily shift representations away from the intended class manifold.

Methodology

The proposed framework enhances standard feature extraction through a two-stage process:

  1. Vector Quantization (VQ): We introduce a learnable "codebook" $E \in R^{n \times d}$ containing feature prototypes. For any extracted feature $z$, the VQ module identifies the nearest prototype $\hat{z}$ in the codebook. By training both clean and corrupted images to map to the same discrete entries, the network learns to treat discretization as a natural denoising filter.
  2. Self-Attention (SA) Fusion: To prevent the loss of fine-grained detail inherent in pure quantization, the original feature $z$ is concatenated with the quantized prototype $\hat{z}$. A Self-Attention module then adaptively fuses these representations, allowing the model to prioritize stable, prototype-based information while retaining necessary high-level details.
Overview of the proposed architecture featuring the Backbone, VQ codebook, and SA module.

The system is optimized using a composite loss function including Cross-Entropy for classification, VQ loss for codebook updates, and a Commitment loss to ensure backbone features remain aligned with the codebook space.

Experimental Results

The method was evaluated on several rigorous benchmarks, including ImageNet-C (corruption robustness), ImageNet-R (generalization to artistic renditions), and ImageNet-A (natural adversarial images).

Robustness and Generalization

On ImageNet-C, the ResNet50-based model achieved a 43.1% mCE, outperforming previous state-of-the-art methods like QualNet. Furthermore, the model showed a 2% accuracy improvement on ImageNet-R, suggesting that quality-independent features also generalize better across different visual domains (e.g., sketches, paintings).

Salience Preservation

Grad-CAM visualizations demonstrate that the VQ-SA module effectively anchors the model's attention. While vanilla models often shift their focus toward background noise or blurred regions in corrupted images, our model consistently maintains focus on the primary object.

Grad-CAM visualizations comparing clean images, QualNet, and our proposed method across various corruptions.

Conclusion

By combining the denoising properties of discrete vector quantization with the adaptive fusion of self-attention, we provide a robust solution for quality-independent representation learning. The module is lightweight and easily integrated into existing architectures. Future work will explore the interaction of these modules with Vision Transformer (ViT) architectures and further optimization of the codebook structure.

Summary updated

Abstract

Recently, the robustness of deep neural networks has drawn extensive attention due to the potential distribution shift between training and testing data (e.g., deep models trained on high-quality images are sensitive to corruption during testing). Many researchers attempt to make the model learn invariant representations from multiple corrupted data through data augmentation or image-pair-based feature distillation to improve the robustness. Inspired by sparse representation in image restoration, we opt to address this issue by learning image-quality-independent feature representation in a simple plug-and-play manner, that is, to introduce discrete vector quantization (VQ) to remove redundancy in recognition models. Specifically, we first add a codebook module to the network to quantize deep features. Then we concatenate them and design a self-attention module to enhance the representation. During training, we enforce the quantization of features from clean and corrupted images in the same discrete embedding space so that an invariant quality-independentfeature representation can be learned to improve the recognition robustness of low-quality images. Qualitative and quantitative experimental results show that our method achieved this goal effectively, leading to a new state-of-the-art result of 43.1 % mCE on ImageNet-C with ResNet50 as the backbone. On other robustness benchmark datasets, such as ImageNet-R, our method also has an accuracy improvement of almost 2%. The source code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/VQSA.htm
BibTeX
@inproceedings{yang2023vector,
  author = {Zhou Yang and Weisheng Dong and Xin Li and Mengluan Huang and Yulin Sun and Guangming Shi},
  title = {{Vector Quantization with Self-Attention for Quality-Independent Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02341},
}
Figure