Contrastive Transformer Masked Image Hashing for Degraded Image Retrieval

Xiaobo Shen, Haoyu Cai, Xiuwen Gong, Yuhui Zheng

Abstract

Large-scale image retrieval systems often face a "degradation gap" between low-quality queries and high-quality database images. This paper introduces Contrastive Transformer Masked Image Hashing (CTMIH), an unsupervised framework that leverages Vision Transformers (ViT) and masked image modeling to learn transform-invariant hash codes. By integrating cross-view debiased contrastive learning and semantic mask reconstruction, CTMIH achieves state-of-the-art performance and remarkable robustness across benchmark datasets like MS COCO, NUS-WIDE, and ImageNet.

Introduction

In modern image retrieval, queries are frequently degraded by blur, noise, or compression, while target databases consist of pristine, high-resolution images. Standard hashing algorithms, which convert images into compact binary codes, are often sensitive to these variations, leading to significant performance drops. While supervised methods can mitigate this, they require extensive labeled data. CTMIH addresses this challenge in an unsupervised manner by learning robust, semantic-preserving features that remain stable across different levels of image degradation.

Methodology

The CTMIH framework utilizes a weight-sharing Siamese network architecture designed to bridge the degradation gap through three primary components:

  1. Vision Transformer (ViT) Backbone: Unlike traditional CNNs, ViTs use self-attention to capture global dependencies, allowing the model to maintain structural understanding even when local patches are degraded.
  2. Cross-view Debiased Contrastive Learning: The model processes two augmented views of an image—one with synthetic degradations and another with random block masking. A debiased contrastive loss ($J_C$) ensures that hash codes for different views of the same image are similar while correcting for potential false negatives in the batch.
  3. Semantic Mask Reconstruction: Inspired by Masked Autoencoders (MAE), the model includes a reconstruction loss ($J_R$) that predicts the latent features (tokens) of masked patches. This forces the network to learn the underlying semantic structure of the image rather than just pixel-level details.
The CTMIH architecture featuring block masking, ViT backbone, and multi-loss optimization.

Experimental Results

The researchers evaluated CTMIH on MS COCO, NUS-WIDE, and ImageNet. To simulate real-world conditions, query images were subjected to synthetic degradations including Gaussian blur, color jitter, and grayscale conversion.

Performance and Robustness

CTMIH consistently outperformed ten state-of-the-art methods. On ImageNet with 64-bit codes, it achieved a Mean Average Precision (mAP) of 0.869, surpassing the next best method, SDC (0.814). A key finding was the model's extreme robustness: while competing models saw performance drops of up to 20% under heavy degradation, CTMIH's mAP remained stable, decreasing by only 1.3%.

mAP performance of CTMIH vs. baselines across increasing levels of image degradation (delta).

Conclusion

CTMIH demonstrates that combining Vision Transformers with masked image modeling and debiased contrastive learning creates a highly effective and robust unsupervised hashing framework. Future work will focus on addressing real-world complex noise, optimizing computational efficiency for mobile deployment, and scaling the approach to massive web-scale datasets.

Summary updated

BibTeX
@inproceedings{shen2024contrastive,
  author = {Xiaobo Shen and Haoyu Cai and Xiuwen Gong and Yuhui Zheng},
  title = {{Contrastive Transformer Masked Image Hashing for Degraded Image Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure