Cross-Modality Perturbation Synergy Attack for Person Re-identification

Yunpeng Gong, Zhun Zhong, Zhiming Luo, Yansong Qu, Rongrong Ji, Min Jiang

Person Re-identification (ReID) serves as the backbone of modern automated surveillance, allowing systems to track individuals across different camera views and timeframes. While much of the existing security research focuses on standard visible-light (RGB) systems, real-world deployments often rely on cross-modality ReID—matching daytime color images with nighttime infrared (IR) or thermal captures.

Recent research presented at NeurIPS 2024 has uncovered a significant vulnerability in these systems. The paper introduces the Cross-Modality Perturbation Synergy (CMPS) attack, the first universal adversarial attack specifically engineered for cross-modality person ReID. By leveraging a novel synergistic optimization strategy, the authors demonstrate that a single noise pattern can effectively blind ReID models across both visible and infrared spectrums.

The Modality Gap Challenge

Cross-modality ReID is inherently more complex than standard ReID due to the "modality gap." An infrared camera captures a grayscale heat map, while an RGB camera captures a full color spectrum. To match these, ReID models must learn to align features that are consistent across both domains.

Existing adversarial attacks—small, intentional perturbations designed to fool AI—are typically "modality-biased." If an attack is optimized for RGB images, it usually fails when the system switches to infrared. The authors of CMPS identified that current attacks lack a mechanism to capture the "shared features" existing across these different data distributions. CMPS aims to bridge this gap, creating a "universal" perturbation that remains effective regardless of whether the camera is seeing in color or heat.

The CMPS Framework

The CMPS framework relies on three primary technical pillars to achieve its high success rate:

  1. Synergistic Optimization: Rather than optimizing for one modality at a time—which often leads to the model "forgetting" how to fool the previous modality—CMPS uses an iterative process that aggregates gradients from both RGB and infrared data. This ensures the resulting noise pattern is effective against the shared feature space the model uses to align the two modalities.
  2. Cross-Modality Triplet Loss: The attack employs a specialized loss function designed to distort the model’s internal "manifold." It pulls "negative" samples (different people) closer to the target in the feature space while pushing "positive" samples (the same person) further away, effectively scrambling the model's ability to distinguish identities.
  3. Cross-Modality Attack Augmentation: To help the attack find the "neutral ground" between color and thermal images, the authors introduce a random grayscale transformation during the attack's training phase. This standardizes the visual representation and forces the perturbation to focus on structural features rather than color-specific ones.

Experimental Evaluation

The researchers evaluated the CMPS attack on three benchmark datasets: SYSU-MM01 (large-scale), RegDB (dual-shot visible/thermal), and LLCM (low-light cross-modality).

The results were striking. When attacking the state-of-the-art AGW model on the SYSU-MM01 dataset, the baseline Rank-1 accuracy of approximately 47.5% was slashed to just 1.11%. In contrast, standard adversarial attacks like PGD only managed to drop the accuracy to roughly 39%.

Furthermore, the attack demonstrated impressive transferability. A perturbation generated to fool one model architecture was often highly effective at fooling completely different architectures without any further tuning. In cross-model tests, the attack maintained success rates as high as 98%.

Key Insights and Future Implications

  • The Power of Grayscale: Using grayscale as a mediator is a highly effective strategy. Since infrared is inherently grayscale, forcing the RGB attack to work on grayscale versions of color images helps the perturbation capture modality-agnostic features.
  • Universal Vulnerability: The attack is "universal" not just across modalities, but across images. A single, static noise pattern can be applied to any pedestrian image to successfully deceive the system, making it a much more practical threat than per-image attacks.
  • The Need for Robustness: The paper primarily focuses on the attack to highlight the fragility of current systems. The authors suggest that "adversarial training"—incorporating these specific cross-modality attacks during the model training phase—could be the key to building more robust and secure surveillance systems in the future.

While CMPS is highly effective, the authors note that gradient-based methods can still struggle with "catastrophic forgetting" when scaled to a massive number of diverse models. Additionally, the current attack assumes the modalities involved (RGB and IR) are known, leaving open questions about how such attacks would perform against unknown sensors like LiDAR or depth cameras.

Summary updated

BibTeX
@inproceedings{gong2024cross,
  author = {Yunpeng Gong and Zhun Zhong and Zhiming Luo and Yansong Qu and Rongrong Ji and Min Jiang},
  title = {{Cross-Modality Perturbation Synergy Attack for Person Re-identification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=LONd7ACEjy},
}
Figure