Abstract
This paper introduces AACH (Adversarial Attack on Deep Cross-Modal Hamming Retrieval), a novel black-box attack framework designed to compromise the integrity of cross-modal retrieval systems. By mapping disparate data modalities into a shared Hamming space, these systems enable efficient similarity searches. AACH disrupts this space by constructing a surrogate model and employing a triplet-based learning strategy to generate visually imperceptible perturbations. These perturbations effectively push query instances away from semantically similar matches and toward unrelated ones. Experimental results on MIRFlickr-25K, NUS-WIDE, and MS COCO demonstrate that AACH significantly degrades retrieval accuracy with high query efficiency, rivaling white-box methods.
Introduction
Deep Cross-Modal Hamming Retrieval (CMHR) has emerged as a critical technology for searching across different data types, such as retrieving images via text queries. By projecting multi-modal data into a compact binary Hamming space, CMHR facilitates rapid and scalable similarity comparisons. However, the reliance on deep learning architectures introduces vulnerabilities to adversarial attacks.
Unlike traditional single-modality classification attacks, attacking cross-modal retrieval requires disrupting the complex similarity structure of a shared embedding space. Furthermore, most existing research assumes a white-box setting, which is often impractical for real-world security assessments. This work addresses the more challenging black-box scenario where the attacker has no knowledge of the target model's internal parameters and must succeed with a limited query budget to avoid detection.
Methodology
The AACH framework operates through three distinct stages to generate effective adversarial examples:
- Surrogate Model Construction: To overcome the black-box constraint, a surrogate model is trained to approximate the target model's behavior. This is achieved by querying the target model with a limited set of data pairs to obtain their corresponding binary codes.
- Triplet Construction Module: This module identifies "positive" (semantically similar) and "negative" (dissimilar) instances for a given query within the Hamming space, forming the basis for the optimization objective.
- Adversarial Perturbation Learning: AACH optimizes a tiny perturbation ($\delta$) using a triplet loss function. This loss pushes the perturbed query away from its true matches and toward unrelated items. A quantization loss is also integrated to ensure the resulting features map effectively to binary codes.
Experimental Evaluation
The effectiveness of AACH was evaluated on three benchmark datasets: MIRFlickr-25K (20,015 pairs), NUS-WIDE (approx. 190,000 pairs), and MS COCO (approx. 120,000 images). For the MS COCO dataset, BERT embeddings were utilized to explore word-level text attacks. A key strength of AACH is its query efficiency, requiring only 1,000 samples to interact with the target models for surrogate training.
Results and Discussion
The primary metric for evaluation was Mean Average Precision (MAP). Key findings include:
- Significant Accuracy Degradation: On the MIRFlickr-25K dataset, AACH reduced the MAP of the state-of-the-art DCMH model from 0.792 to 0.631 for image-to-text retrieval.
- Black-Box Performance: AACH achieved performance comparable to leading white-box attacks like CMLA without requiring internal model access or label information.
- Asymmetric Vulnerability: Adversarial images were found to be generally more effective than adversarial text, likely due to the higher dimensionality and continuous nature of the image space.
- Transferability: Perturbations exhibited significant transferability across different code lengths, suggesting fundamental vulnerabilities in how semantic correlations are learned in Hamming space.
Conclusion
AACH demonstrates a potent threat to cross-modal retrieval systems in realistic black-box scenarios. By manipulating the neighborhood of a query rather than just flipping a label, it systematically reorganizes search results to be irrelevant. While highly efficient, the requirement of a query budget remains a potential point of detection. Future research should focus on more sophisticated text-generation attacks and the development of robust hashing methods to defend against triplet-based adversarial strategies.
Summary updated