Overview
This paper introduces CMLA (Cross-Modal correlation Learning with Adversarial samples), the first framework to demonstrate that cross-modal hashing systems—used to retrieve text from images and vice versa—are highly vulnerable to adversarial attacks. By crafting "deceptive" perturbations that break cross-modal links while preserving single-modality consistency, the authors show how to both fool these systems and significantly harden them through adversarial training.
The Vulnerability of Cross-Modal Retrieval
Deep neural networks have revolutionized cross-modal retrieval, such as searching for a photo using a text query. To make this efficient at scale, researchers use cross-modal hashing, which maps different data types (images and text) into a shared binary "hash" space where similar items have similar codes.
However, deep networks are notoriously brittle. A tiny, invisible change to an image (an adversarial perturbation) can make a classifier see a "cat" as a "toaster." While this has been studied extensively in single-modality tasks like image classification, the authors identify a major gap: Is cross-modal retrieval also vulnerable? If an attacker can subtly alter an image so it no longer retrieves the correct text, or vice versa, it poses a significant security risk for search engines and safety-critical systems.
Methods: Crafting a "Deceptive" Attack
The authors propose CMLA, a method to generate adversarial samples specifically tailored for cross-modal hashing.
A naive attack might just scramble the hash code, but that could make the image look like "noise" to a single-modality system, making the attack easy to detect. CMLA avoids this by using two simultaneous regularizations during optimization:
- Inter-Modality Similarity Regularization: This pushes the hash code of the perturbed sample (e.g., an image) away from its corresponding pair in the other modality (e.g., the text). This effectively "breaks the bridge" between modalities.
- Intra-Modality Similarity Regularization: Crucially, this ensures the perturbed sample still maintains its relationship with other samples within its own modality.
By balancing these two forces, CMLA creates an adversarial image that still "looks" and "acts" like a normal image to an image-only system but fails completely when used to search for text. The optimization is performed iteratively to find the smallest possible perturbation that achieves this goal.
Key Insights: Breaking and Fixing the Space
The most important insight of CMLA is that a "good" cross-modal attack shouldn't just be invisible to the human eye; it should be invisible to single-modality checks. If an adversarial image still retrieves similar images correctly but fails to retrieve the right text, it is much harder to detect.
Furthermore, the authors demonstrate that these adversarial samples aren't just for attacking; they are the "medicine." By including CMLA-generated samples in the training set, the resulting model becomes significantly more robust against future attacks. The adversarial training forces the model to redraw its decision boundaries, effectively "fixing the bridge" that the attack attempted to break.
Key Results
The researchers evaluated their approach on two gold-standard benchmarks for cross-modal retrieval: MIRFlickr-25K and NUS-WIDE.
- Effectiveness of Attack: CMLA successfully "fooled" state-of-the-art models like DCMH and SSAH. For example, on the MIRFlickr-25K dataset, the Mean Average Precision (MAP) for image-to-text retrieval on a DCMH model dropped from 0.749 to 0.521 after 500 iterations of CMLA.
- Imperceptibility: The distortions required to achieve these drops were extremely low (often < 0.03), meaning the changes were virtually invisible to human eyes.
- Improved Robustness: After performing adversarial training with CMLA samples, the models' performance under attack improved dramatically. In some cases, the "hardened" models even outperformed the original "clean" models on standard retrieval tasks, suggesting that adversarial samples help the network explore the search space more thoroughly.
Limitations and Future Work
- Text Modality Constraints: While the paper addresses text, the perturbations are applied to bag-of-words vectors. Modern NLP uses transformers and word embeddings; extending CMLA to these more complex, sequential text models is a natural next step.
- Computational Cost: Generating adversarial samples for every training point to perform adversarial training is computationally expensive, which might limit scalability for massive datasets.
- Black-Box Attacks: The current study focuses on "white-box" attacks (where the attacker knows the model's parameters). Future work could explore "black-box" scenarios where the attacker only has access to the model's outputs.
Summary updated