Steal Now and Attack Later: Evaluating Robustness of Object Detection against Black-box Adversarial Attacks

Erh-Chung Chen, Pin-Yu Chen, I-Hsin Chung, Che-Rung LeearXiv 2024 · OpenReview

Abstract

Latency attacks against object detection represent a variant of adversarial attacks that aim to inflate the inference time by generating additional ghost objects in a target image. However, generating ghost objects in the black-box scenario remains a challenge since information about these unqualified objects remains opaque. In this study, we demonstrate the feasibility of generating ghost objects in adversarial examples by extending the concept of "steal now, decrypt later" attacks. These adversarial examples, once produced, can be employed to exploit potential vulnerabilities in the AI service, giving rise to significant security concerns. The experimental results demonstrate that the proposed attack achieves successful attacks across various commonly used models and Google Vision API without any prior knowledge about the target model. Additionally, the average cost of each attack is less than $1 dollars, posing a significant threat to AI security.

Introduction

Adversarial research in computer vision has traditionally focused on evasion attacks—making objects disappear—or misclassification attacks—altering an object's perceived identity. However, a more insidious threat is emerging: Latency Attacks. Unlike traditional attacks that aim to deceive human perception, latency attacks target the computational resources of the AI system.

In object detection, processing time is often directly proportional to the number of objects detected. By injecting numerous "ghost objects"—perturbations that appear as noise to humans but are recognized as valid entities by the model—an attacker can force the system into computational exhaustion. This can lead to significant lag, timeouts, or complete system crashes. Executing such attacks in a "black-box" setting, where the target model's architecture is unknown, presents a significant challenge as the attacker must identify universal triggers for detection without direct feedback.

Methodology: The "Steal Now, Attack Later" Framework

The researchers propose a novel three-stage strategy designed to overcome the limitations of black-box environments while maintaining low query costs.

The proposed 'Steal Now, Attack Later' pipeline, consisting of data collection, candidate selection, and color projection stages.
  1. Data Collection (The "Steal"): To avoid the high costs of querying commercial APIs, the attacker first "steals" potential object features from public datasets like MS COCO or Open Images. By using a cheap, locally-deployed surrogate model, they curate a database of image patches that are highly likely to be recognized as objects by various detection architectures.
  2. Position-centric Candidate Selection: Efficiency is achieved through a grid-based placement strategy. Instead of testing patches individually, the target image is divided into a grid. Multiple "ghost" patches are tested across these cells simultaneously, maximizing the number of potential detections per query and significantly reducing the search space.
  3. Color Manipulation and Projection: To maintain stealth, the added perturbations must remain within a defined mathematical limit (the epsilon ball). The authors utilize a projection algorithm that minimizes the visual intensity of the noise while ensuring that the underlying features remain detectable by the AI's feature extraction layers.

Experimental Setup

The study evaluated the attack's effectiveness using industry-standard benchmarks:

  • Datasets: MS COCO 2017 and Open Images were used for both harvesting candidate patches and providing target images for evaluation. PASCAL VOC served as an additional source for labeled object data.
  • Target Models: The attack was tested against a variety of state-of-the-art architectures, including YOLOv8, Faster R-CNN, and DETR.
  • Commercial APIs: To test real-world applicability, the researchers targeted public cloud services, specifically the Google Cloud Vision API.

Results and Discussion

The primary metric for success was the "Attack Success Rate" (ASR), defined as the ability to increase the detected object count by more than 20.

Visualization of the attack process: (a) original image, (b-d) adversarial perturbations at different stages, (e) original detections, and (f) detections on the adversarial image showing numerous 'ghost' objects.

Model Vulnerabilities

YOLOv8 proved to be the most susceptible model, reaching an ASR of 83% under moderate perturbation levels ($\epsilon=32$). More impressively, the attack successfully bypassed the Google Cloud Vision API with an ASR of 18%, despite having zero prior knowledge of Google's internal configurations. This demonstrates a high degree of adversarial transferability.

The Economics of Attack

A critical finding is the low cost of execution. The researchers estimate that a successful attack on a commercial API costs less than $1 USD. This low barrier to entry transforms latency attacks from a theoretical concern into a practical, high-stakes threat for AI service providers.

Accidental Robustness and Limitations

Interestingly, some models like SSD300 and Azure Vision exhibited "accidental robustness." These systems often include internal filters that discard very small detections. Since the proposed attack relies on many small "ghost" patches, these filters effectively neutralized the threat. Additionally, input resizing by cloud services (e.g., downscaling to 64x64) can "wash out" fine-tuned noise, providing another layer of defense.

Conclusion

As AI systems are increasingly integrated into real-time, safety-critical applications like autonomous driving and automated surveillance, the threat of computational exhaustion becomes paramount. This research highlights that current object detectors suffer from "contextual blindness," dutifully processing nonsensical clusters of objects that a human would immediately flag. Future defenses must move beyond pixel-level robustness and incorporate semantic context checkers to identify and mitigate these low-cost, high-impact latency attacks.

Summary updated

Abstract

Latency attacks against object detection represent a variant of adversarial attacks that aim to inflate the inference time by generating additional ghost objects in a target image. However, generating ghost objects in the black-box scenario remains a challenge since information about these unqualified objects remains opaque. In this study, we demonstrate the feasibility of generating ghost objects in adversarial examples by extending the concept of "steal now, decrypt later" attacks. These adversarial examples, once produced, can be employed to exploit potential vulnerabilities in the AI service, giving rise to significant security concerns. The experimental results demonstrate that the proposed attack achieves successful attacks across various commonly used models and Google Vision API without any prior knowledge about the target model. Additionally, the average cost of each attack is less than \$ 1 dollars, posing a significant threat to AI security.
BibTeX
@misc{chen2024steal,
  author = {Erh-Chung Chen and Pin-Yu Chen and I-Hsin Chung and Che-Rung Lee},
  title = {{Steal Now and Attack Later: Evaluating Robustness of Object Detection against Black-box Adversarial Attacks}},
  year = {2024},
  eprint = {2404.15881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.15881},
}
Figure