Zero-Query Transfer Attacks on Context-Aware Object Detectors

Zikui Cai, Shantanu Rane, Alejandro E. Brito, Chengyu Song, Srikanth V. Krishnamurthy, Amit K. Roy-Chowdhury, M. Salman Asif

Introduction

Deep neural networks (DNNs) have revolutionized object detection but remain vulnerable to adversarial perturbations—subtle pixel changes that lead to misclassification. To counter this, "context-aware" detectors have been developed. These systems evaluate the logical consistency of a scene; for instance, a "boat" appearing on a "crosswalk" next to a "stop sign" is flagged as an anomaly. While effective against naive attacks, these defenses introduce a new attack surface.

The Challenge of Contextual Consistency

Traditional black-box attacks often rely on multiple queries to a victim model to refine perturbations. However, high query volumes are easily detected by traffic monitoring. The core challenge is to craft a "zero-query" attack that is both effective and "context-consistent," ensuring the final perturbed scene appears logical to a context-aware defense mechanism.

Comparison of original, context-inconsistent, and context-consistent attacks.

As shown in Figure 1, a context-inconsistent attack might turn a "crosswalk" into a "boat," which is easily detected. A context-consistent attack, however, holistically alters the scene (e.g., turning the "crosswalk" into "water" and the "man" into a "boat") to maintain logical coherence.

Methodology: The ZQA-PSPM Framework

The researchers propose the Zero-Query Attack (ZQA) framework, which shifts the focus from attacking individual objects to attacking entire scenes.

  1. Context Modeling: A co-occurrence graph is constructed from training data (e.g., Pascal VOC) to define which object labels logically belong together.
  2. Perturbation Success Probability Matrix (PSPM): This is the framework's key innovation. Using a surrogate model, the attacker pre-calculates the likelihood of successful label swaps (e.g., dog to cat) under specific perturbation budgets.
  3. Holistic Attack Planning: Instead of random changes, the algorithm identifies a target object and uses the PSPM and Context Graph to select "bystander" objects to perturb. This ensures the final set of labels forms a consistent subgraph in the context model.
  4. Execution: The attack is implemented using Projected Gradient Descent (PGD), guided by the holistic plan to ensure the entire scene is transformed convincingly.

Experimental Evaluation

The ZQA-PSPM approach was evaluated on Pascal VOC 2007 and MS COCO benchmarks, focusing on images with 2–6 objects to ensure context was a significant factor.

Key Results

The proposed method demonstrated significant improvements over existing baselines.

Performance comparison of ZQA-PSPM against context-agnostic and few-query attacks.

As detailed in Figure 6, the ZQA-PSPM approach consistently achieved higher fooling rates than context-agnostic methods across various perturbation budgets ($\epsilon$). Notably, the zero-query attack often outperformed traditional "few-query" attacks that were allowed up to three rounds of feedback from the victim model. Furthermore, the attacks showed strong transferability across different architectures, such as from Faster R-CNN to RetinaNet or FoveaBox.

Conclusion and Future Directions

This research highlights a critical vulnerability: context-aware defenses can be bypassed without any direct interaction with the victim model. While the current approach relies on co-occurrence, future work could incorporate spatial relationships and relative object sizes to create even more sophisticated "lies." This work serves as a call to action for developing more robust, multi-modal defensive strategies in computer vision.

Summary updated

Abstract

Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check, wherein, if the detected objects are not consistent with an appropriately defined context, then an attack is suspected. Stronger attacks are needed to fool such context-aware detectors. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check of black-box object detectors operating on complex, natural scenes. Unlike many black-box attacks that perform repeated attempts and open themselves to detection, we assume a "zero-query" setting, where the attacker has no knowledge of the classification decisions of the victim system. First, we derive multiple attack plans that assign incorrect labels to victim objects in a context-consistent manner. Then we design and use a novel data structure that we call the perturbation success probability matrix, which enables us to filter the attack plans and choose the one most likely to succeed. This final attack plan is implemented using a perturbation-bounded adversarial attack algorithm. We compare our zero-query attack against a few-query scheme that repeatedly checks if the victim system is fooled. We also compare against state-of-the-art context-agnostic attacks. Against a context-aware defense, the fooling rate of our zero-query approach is significantly higher than context-agnostic approaches and higher than that achievable with up to three rounds of the few-query scheme.
BibTeX
@inproceedings{cai2022zero,
  author = {Zikui Cai and Shantanu Rane and Alejandro E. Brito and Chengyu Song and Srikanth V. Krishnamurthy and Amit K. Roy-Chowdhury and M. Salman Asif},
  title = {{Zero-Query Transfer Attacks on Context-Aware Object Detectors}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01460},
}
Figure