Abstract
Traditional adversarial attacks rely on mathematical $L_p$ norms as proxies for human perception, which often fail to capture true visual similarity. This paper introduces a "perception-in-the-loop" framework using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to incorporate actual human judgment into the adversarial generation process. The results show that human-guided black-box optimization can efficiently discover perturbations that are invisible to people but catastrophic for AI models, even those employing state-of-the-art adversarial defenses.
The Problem with $L_p$ Norms
Most research into adversarial examples—small perturbations that cause a model to misclassify an input—relies on $L_p$ norms (like $L_2$ or $L_\infty$) as a proxy for human perception. The assumption is that if the mathematical distance between an original image and an adversarial one is small, a human won't notice the difference.
However, $L_p$ norms are notoriously poor reflections of how humans actually perceive similarity. An image can have a large $L_p$ distance but look identical to a human, or have a tiny $L_p$ distance but look obviously corrupted. The authors argue that to truly understand and defend against adversarial attacks, we must move beyond these mathematical proxies and incorporate actual human perception into the attack generation process.
Methodology: Perception-in-the-Loop
The authors propose using Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a derivative-free, black-box optimization technique. Unlike many optimization algorithms that require a precise numerical cost or gradient, CMA-ES only requires a ranking of candidates.
The Human-in-the-Loop Process: 1. Generation: CMA-ES generates a population of perturbed images. 2. Filtering: Any image that is not misclassified by the target model is hidden. 3. Human Selection: A human participant is shown the remaining misclassified images and asked to pick the "top $K$" images that look most like the original. 4. Update: CMA-ES uses this ranking to update its internal distribution and generate the next, even more "perceptually similar" generation.
Optimization and Refinement
Humans are much better at providing relative preferences ("Image A looks more like the original than Image B") than providing absolute quantitative scores. CMA-ES is uniquely suited for this because it only cares about the top-tier candidates, not their absolute values. To further refine the results, the authors use a bisection method as a post-processing step to minimize the perturbation once a misclassification is found.
Key Results and the "Robustness Illusion"
The method was evaluated on MNIST, CIFAR-10, and the German Traffic Sign Recognition Benchmark (GTSRB). Key findings include:
- Efficiency: Adversarial examples were found with very small populations (as few as 4) and few iterations (~10), making human interaction feasible.
- Perceptual Consistency: Human perceptual preferences were remarkably robust across different individuals, providing a stable signal for optimization.
- Breaking "Robust" Models: Models trained specifically to be robust against $L_\infty$ perturbations were still highly susceptible. While a standard FGSM attack only succeeded 36.7% of the time against such models, the perception-in-the-loop attack achieved an 83.3% success rate.
Conclusion
The study suggests that current defenses are over-fitting to specific mathematical norms rather than achieving true perceptual robustness. By using human-in-the-loop data, researchers may eventually develop better mathematical models of human perception that can replace the human in the loop for large-scale adversarial training.
Summary updated