TL;DR
This paper introduces AutoLoss-AR, a novel AutoML approach to automatically discover surrogate loss functions for evaluating adversarial risk in deep neural networks. By optimizing the gap between true and approximated adversarial risk, AutoLoss-AR finds more effective loss functions than manually designed ones, leading to more accurate robustness assessments.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are notoriously vulnerable to adversarial examples – small, imperceptible perturbations that can drastically alter a model's predictions. Accurately evaluating a DNN's robustness against these attacks is crucial for deploying them in security-sensitive applications. Adversarial risk is a key metric for this evaluation, but its exact calculation is computationally intractable because it involves optimizing a discontinuous 0-1 loss function. Current methods approximate this risk using surrogate loss functions (like Cross-Entropy or Difference of Logits Ratio), but these are manually designed and may not be optimal for capturing the true adversarial risk. This can lead to a misleading sense of robustness. The paper aims to bridge this gap by automatically searching for better surrogate loss functions that minimize the approximation error between the true adversarial risk and its computed approximation.
Methods
The core of the paper is AutoLoss-AR, an adaptation of the AutoLoss-Zero framework for adversarial risk evaluation. AutoLoss-AR employs Genetic Programming (GP) to search for optimal surrogate loss functions. The process involves:
- Search Space Design: A custom search space is created, including primitive operations (like
exp,max,softmax,add,mul) and input nodes representing model logits and true labels. This space is informed by existing handcrafted loss functions. - Fitness Evaluation: The fitness of each candidate loss function is its ability to maximize the approximated adversarial risk (Equation 7). This is evaluated using the Projected Gradient Descent (PGD) attack (specifically PGD-10 for efficiency during search) on a subset of the data.
- GP Algorithm: The GP algorithm iteratively evolves a population of candidate loss functions through steps of initialization, evaluation, selection (tournament selection), crossover, and mutation.
- Distillation of Losses: After the search, the discovered complex tree-encoded loss functions are simplified into clean, readable formulas (e.g., BS1-BS5) for practical use.
Datasets
The experiments were conducted using the CIFAR-10 and CIFAR-100 datasets.
Key Insights and Ideas
- AutoML for Robustness Evaluation: The paper demonstrates that AutoML techniques, specifically GP, can be effectively applied to discover optimal loss functions for evaluating adversarial robustness, going beyond manual design.
- Tightening Approximation Error: By directly optimizing to minimize the gap between true and approximated adversarial risk, AutoLoss-AR yields more accurate robustness assessments than traditional methods.
- Generalizability of Discovered Losses: The simplified, discovered loss functions (BS1-BS5) show good performance even on unseen adversarial training models, suggesting they can serve as valuable, reusable knowledge for robustness evaluation.
- Local Consistency Matters: Visualizing the loss landscapes reveals that the best-performing discovered losses exhibit better local consistency with the true 0-1 loss and are easier for optimization algorithms (like PGD) to navigate, leading to better adversarial example generation.
Key Results
- White-box Evaluation: Using AutoLoss-AR on 10 adversarially trained models, the best-discovered losses achieved adversarial accuracies that were 0.2% to 1.6% better than those obtained with handcrafted baselines.
- Black-box Evaluation: When tested on 7 unseen adversarially trained models, the 5 distilled surrogate losses outperformed handcrafted baselines by 0.8% to 2.4% in terms of adversarial accuracy.
- Search Efficiency: In independent searches, AutoLoss-AR successfully found better-performing losses than handcrafted baselines in a significant majority of runs (e.g., 9/15, 12/15 for different models).
- Qualitative Examples: Generated adversarial examples using the discovered BS5 loss were visually similar to clean images, indicating their effectiveness in fooling models.
Limitations and Future Work
- Computational Cost: While efforts were made to reduce computation (e.g., PGD-10 during search, evaluating on 1000 images), the search process can still be computationally intensive.
- Hyperparameter Tuning: The performance of AutoLoss-AR is dependent on its own hyperparameters (e.g., population size, mutation rate), which require careful tuning.
- Future Work: The authors suggest exploring more complex search spaces, different optimization algorithms for fitness evaluation, and applying AutoLoss-AR to other domains beyond image classification. Further investigation into the theoretical properties of the discovered losses and their relationship with different attack algorithms is also warranted.
Summary updated