Adversarial Attack on Deep Cross-Modal Hamming Retrieval

Chao Li, Shangqian Gao, Cheng Deng, Wei Liu, Heng Huang

Abstract

This paper introduces AACH (Adversarial Attack on Deep Cross-Modal Hamming Retrieval), a novel black-box attack framework designed to compromise the integrity of cross-modal retrieval systems. By mapping disparate data modalities into a shared Hamming space, these systems enable efficient similarity searches. AACH disrupts this space by constructing a surrogate model and employing a triplet-based learning strategy to generate visually imperceptible perturbations. These perturbations effectively push query instances away from semantically similar matches and toward unrelated ones. Experimental results on MIRFlickr-25K, NUS-WIDE, and MS COCO demonstrate that AACH significantly degrades retrieval accuracy with high query efficiency, rivaling white-box methods.

Introduction

Deep Cross-Modal Hamming Retrieval (CMHR) has emerged as a critical technology for searching across different data types, such as retrieving images via text queries. By projecting multi-modal data into a compact binary Hamming space, CMHR facilitates rapid and scalable similarity comparisons. However, the reliance on deep learning architectures introduces vulnerabilities to adversarial attacks.

Overview of Cross-Modal Hamming Retrieval and the conceptual mechanism of the proposed adversarial attack.

Unlike traditional single-modality classification attacks, attacking cross-modal retrieval requires disrupting the complex similarity structure of a shared embedding space. Furthermore, most existing research assumes a white-box setting, which is often impractical for real-world security assessments. This work addresses the more challenging black-box scenario where the attacker has no knowledge of the target model's internal parameters and must succeed with a limited query budget to avoid detection.

Methodology

The AACH framework operates through three distinct stages to generate effective adversarial examples:

  1. Surrogate Model Construction: To overcome the black-box constraint, a surrogate model is trained to approximate the target model's behavior. This is achieved by querying the target model with a limited set of data pairs to obtain their corresponding binary codes.
  2. Triplet Construction Module: This module identifies "positive" (semantically similar) and "negative" (dissimilar) instances for a given query within the Hamming space, forming the basis for the optimization objective.
  3. Adversarial Perturbation Learning: AACH optimizes a tiny perturbation ($\delta$) using a triplet loss function. This loss pushes the perturbed query away from its true matches and toward unrelated items. A quantization loss is also integrated to ensure the resulting features map effectively to binary codes.
The AACH framework, illustrating surrogate model construction, triplet construction, and adversarial example learning.

Experimental Evaluation

The effectiveness of AACH was evaluated on three benchmark datasets: MIRFlickr-25K (20,015 pairs), NUS-WIDE (approx. 190,000 pairs), and MS COCO (approx. 120,000 images). For the MS COCO dataset, BERT embeddings were utilized to explore word-level text attacks. A key strength of AACH is its query efficiency, requiring only 1,000 samples to interact with the target models for surrogate training.

Results and Discussion

The primary metric for evaluation was Mean Average Precision (MAP). Key findings include:

  • Significant Accuracy Degradation: On the MIRFlickr-25K dataset, AACH reduced the MAP of the state-of-the-art DCMH model from 0.792 to 0.631 for image-to-text retrieval.
  • Black-Box Performance: AACH achieved performance comparable to leading white-box attacks like CMLA without requiring internal model access or label information.
  • Asymmetric Vulnerability: Adversarial images were found to be generally more effective than adversarial text, likely due to the higher dimensionality and continuous nature of the image space.
  • Transferability: Perturbations exhibited significant transferability across different code lengths, suggesting fundamental vulnerabilities in how semantic correlations are learned in Hamming space.

Conclusion

AACH demonstrates a potent threat to cross-modal retrieval systems in realistic black-box scenarios. By manipulating the neighborhood of a query rather than just flipping a label, it systematically reorganizes search results to be irrelevant. While highly efficient, the requirement of a query budget remains a potential point of detection. Future research should focus on more sophisticated text-generation attacks and the development of robust hashing methods to defend against triplet-based adversarial strategies.

Summary updated

BibTeX
@inproceedings{li2021adversarial,
  author = {Chao Li and Shangqian Gao and Cheng Deng and Wei Liu and Heng Huang},
  title = {{Adversarial Attack on Deep Cross-Modal Hamming Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00222},
}
Figure