Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks

Quang H. Nguyen, Yingjie Lao, Tung Pham, Kok-Seng Wong, Khoa D. Doan

TL;DR

This paper introduces a novel defense mechanism against adversarial attacks on deep neural networks by injecting random noise into the model's hidden features during inference. This "randomized feature defense" significantly improves robustness against query-based black-box attacks without requiring adversarial training, offering a practical and lightweight solution for protecting pre-trained models.

What Problem Does This Paper Address?

Deep neural networks, despite their impressive performance, are notoriously vulnerable to adversarial attacks. These attacks involve subtly altering input data (like images) in ways imperceptible to humans, yet capable of causing the model to misclassify them. While white-box attacks, which have full access to the model's internals, are well-studied, a more practical threat comes from black-box attacks. In these scenarios, attackers only have access to the model's output (e.g., class probabilities or final predictions) and must iteratively query the model to craft adversarial examples. This paper focuses on defending against these query-based black-box attacks, which are particularly challenging because they don't require knowledge of the model's architecture or parameters. The core problem is how to make models resilient to these attacks without sacrificing accuracy or requiring computationally expensive retraining.

Methods

The proposed defense, termed "Randomized Feature Defense," injects random noise into the hidden feature representations at intermediate layers of a neural network during inference. The key idea is to make the model's internal state "stochastic" from the attacker's perspective. When an attacker queries the model, they receive a slightly different output each time due to this injected noise. This makes it much harder for the attacker to accurately estimate the gradient of the loss function with respect to the input, which is crucial for many query-based attack strategies (like score-based attacks that rely on gradient estimation or decision-based attacks that search for decision boundaries).

The defense is "plug-and-play," meaning it can be applied to any pre-trained model without retraining. The noise is typically sampled from a Gaussian distribution. The paper also theoretically analyzes the importance of where this noise is injected, suggesting that strategically choosing layers based on the gradient of the adversarial objective function can further enhance robustness. This analysis reveals that the defense's effectiveness is linked to the ratio of defense noise to attack noise and the relative magnitudes of gradients with respect to features versus input.

Datasets

The experiments were conducted on two widely used benchmark datasets for adversarial robustness:

  • CIFAR10: A dataset of small color images in 10 classes.
  • ImageNet: A large-scale dataset of diverse images across 1000 classes.

For experiments, a subset of 1000 images was randomly selected from each dataset, ensuring representation from every class.

Key Insights and Ideas

  • Feature-space noise is effective: Injecting noise into intermediate features, rather than just the input, is a powerful way to disrupt gradient estimation for black-box attacks. This is because it directly perturbs the information flow within the network.
  • No adversarial training needed: A significant advantage is that this defense doesn't require the computationally intensive and often accuracy-degrading process of adversarial training. This makes it highly practical for deploying on existing models.
  • Strategic noise injection matters: The paper's theoretical analysis highlights that the choice of which layers to perturb and the amount of noise are critical. This suggests that defenses can be tailored to specific attack types or model architectures.
  • Robustness-vs-Accuracy Trade-off: While the defense improves robustness, there's a slight trade-off with clean accuracy. The paper emphasizes that their defense aims for a minimal drop in accuracy, making it a practical choice.
  • Decision-based attacks are still challenging: While the defense improves robustness against decision-based attacks, they remain a significant threat, especially when the attacker can adapt their strategy.

Key Results

  • Significant Robustness Improvement: The randomized feature defense consistently and significantly improves the robustness of models against various score-based (NES, Square, SignHunt) and decision-based (RayS, SignFlip) black-box attacks across different architectures (VGG19, ResNet50, ViT, DeiT) on both ImageNet and CIFAR10.
  • Competitive Performance: In many cases, the proposed defense outperforms other randomized input defenses, achieving higher robust accuracy with comparable clean accuracy drops. For example, on VGG19 against Square attack on ImageNet, the feature defense achieved 22.2% robust accuracy compared to 17.8% for input defense.
  • Minimal Accuracy Impact: The defense typically results in a small drop in clean accuracy (around 1-2%), making it a practical choice for real-world applications.
  • Theoretical Validation: The empirical results align with the theoretical analysis, demonstrating that higher noise ratios and specific gradient characteristics lead to increased robustness.

Limitations and Future Work

  • Adaptive Attacks: The paper acknowledges that adaptive attacks, which specifically try to counter the defense (e.g., using Expectation Over Transformation - EOT), can mitigate the effectiveness of randomized defenses. While their defense still shows better performance, it's not a complete solution against such sophisticated attacks.
  • Future Work: The authors suggest future research directions, including:
  • Developing defenses that can handle black-box attacks that transfer adversarial samples from a surrogate model to the target model.
  • Further analysis of defenses against more complex black-box attack scenarios.

Summary updated

Abstract

Recent works have shown that deep neural networks are vulnerable to adversarial examples that find samples close to the original image but can make the model misclassify. Even with access only to the model's output, an attacker can employ black-box attacks to generate such adversarial examples. In this work, we propose a simple and lightweight defense against black-box attacks by adding random noise to hidden features at intermediate layers of the model at inference time. Our theoretical analysis confirms that this method effectively enhances the model's resilience against both score-based and decision-based black-box attacks. Importantly, our defense does not necessitate adversarial training and has minimal impact on accuracy, rendering it applicable to any pre-trained model. Our analysis also reveals the significance of selectively adding noise to different parts of the model based on the gradient of the adversarial objective function, which can be varied during the attack. We demonstrate the robustness of our defense against multiple black-box attacks through extensive empirical experiments involving diverse models with various architectures.
BibTeX
@inproceedings{nguyen2024understanding,
  author = {Quang H. Nguyen and Yingjie Lao and Tung Pham and Kok-Seng Wong and Khoa D. Doan},
  title = {{Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=vZ6r9GMT1n},
}
Figure