Perception-in-the-Loop Adversarial Examples

Mahmoud Salamati, Sadegh Soudjani, Rupak MajumdararXiv 2019 · arXiv

Abstract

Traditional adversarial attacks rely on mathematical $L_p$ norms as proxies for human perception, which often fail to capture true visual similarity. This paper introduces a "perception-in-the-loop" framework using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to incorporate actual human judgment into the adversarial generation process. The results show that human-guided black-box optimization can efficiently discover perturbations that are invisible to people but catastrophic for AI models, even those employing state-of-the-art adversarial defenses.

The Problem with $L_p$ Norms

Most research into adversarial examples—small perturbations that cause a model to misclassify an input—relies on $L_p$ norms (like $L_2$ or $L_\infty$) as a proxy for human perception. The assumption is that if the mathematical distance between an original image and an adversarial one is small, a human won't notice the difference.

However, $L_p$ norms are notoriously poor reflections of how humans actually perceive similarity. An image can have a large $L_p$ distance but look identical to a human, or have a tiny $L_p$ distance but look obviously corrupted. The authors argue that to truly understand and defend against adversarial attacks, we must move beyond these mathematical proxies and incorporate actual human perception into the attack generation process.

Methodology: Perception-in-the-Loop

The authors propose using Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a derivative-free, black-box optimization technique. Unlike many optimization algorithms that require a precise numerical cost or gradient, CMA-ES only requires a ranking of candidates.

The perception-in-the-loop framework overview, showing the iterative cycle between the CMA-ES algorithm, model filtering, and human selection.

The Human-in-the-Loop Process: 1. Generation: CMA-ES generates a population of perturbed images. 2. Filtering: Any image that is not misclassified by the target model is hidden. 3. Human Selection: A human participant is shown the remaining misclassified images and asked to pick the "top $K$" images that look most like the original. 4. Update: CMA-ES uses this ranking to update its internal distribution and generate the next, even more "perceptually similar" generation.

Optimization and Refinement

Humans are much better at providing relative preferences ("Image A looks more like the original than Image B") than providing absolute quantitative scores. CMA-ES is uniquely suited for this because it only cares about the top-tier candidates, not their absolute values. To further refine the results, the authors use a bisection method as a post-processing step to minimize the perturbation once a misclassification is found.

Geometric representation of the optimization process, illustrating how candidate points (x1, x2, x3) are ranked to find an optimal adversarial example (x*) relative to the reference image (xref).

Key Results and the "Robustness Illusion"

The method was evaluated on MNIST, CIFAR-10, and the German Traffic Sign Recognition Benchmark (GTSRB). Key findings include:

  • Efficiency: Adversarial examples were found with very small populations (as few as 4) and few iterations (~10), making human interaction feasible.
  • Perceptual Consistency: Human perceptual preferences were remarkably robust across different individuals, providing a stable signal for optimization.
  • Breaking "Robust" Models: Models trained specifically to be robust against $L_\infty$ perturbations were still highly susceptible. While a standard FGSM attack only succeeded 36.7% of the time against such models, the perception-in-the-loop attack achieved an 83.3% success rate.

Conclusion

The study suggests that current defenses are over-fitting to specific mathematical norms rather than achieving true perceptual robustness. By using human-in-the-loop data, researchers may eventually develop better mathematical models of human perception that can replace the human in the loop for large-scale adversarial training.

Summary updated

Abstract

We present a scalable, black box, perception-in-the-loop technique to find adversarial examples for deep neural network classifiers. Black box means that our procedure only has input-output access to the classifier, and not to the internal structure, parameters, or intermediate confidence values. Perception-in-the-loop means that the notion of proximity between inputs can be directly queried from human participants rather than an arbitrarily chosen metric. Our technique is based on covariance matrix adaptation evolution strategy (CMA-ES), a black box optimization approach. CMA-ES explores the search space iteratively in a black box manner, by generating populations of candidates according to a distribution, choosing the best candidates according to a cost function, and updating the posterior distribution to favor the best candidates. We run CMA-ES using human participants to provide the fitness function, using the insight that the choice of best candidates in CMA-ES can be naturally modeled as a perception task: pick the top $k$ inputs perceptually closest to a fixed input. We empirically demonstrate that finding adversarial examples is feasible using small populations and few iterations. We compare the performance of CMA-ES on the MNIST benchmark with other black-box approaches using $L_p$ norms as a cost function, and show that it performs favorably both in terms of success in finding adversarial examples and in minimizing the distance between the original and the adversarial input. In experiments on the MNIST, CIFAR10, and GTSRB benchmarks, we demonstrate that CMA-ES can find perceptually similar adversarial inputs with a small number of iterations and small population sizes when using perception-in-the-loop. Finally, we show that networks trained specifically to be robust against $L_\infty$ norm can still be susceptible to perceptually similar adversarial examples.
BibTeX
@misc{salamati2019perception,
  author = {Mahmoud Salamati and Sadegh Soudjani and Rupak Majumdar},
  title = {{Perception-in-the-Loop Adversarial Examples}},
  year = {2019},
  eprint = {1901.06834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.06834},
}
Figure