Theoretically Grounded Loss Functions and Algorithms for Adversarial Robustness

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, Yutao ZhongAISTATS 2023 · OpenReview

Abstract

Adversarial robustness is a critical property of classifiers in applications as they are increasingly deployed in complex real-world systems. Yet, achieving accurate adversarial robustness in machine learning remains a significant challenge. This work introduces a new family of "smooth adversarial losses" that provide strong theoretical guarantees ($$-consistency) for adversarial training. We translate this theory into a practical algorithm called **PSAL** (Principled Smooth Adversarial Loss), which improves both robust and clean accuracy across standard benchmarks. ## Introduction: The Robustness Challenge Deep learning models are notoriously fragile; tiny, imperceptible adversarial perturbations can cause state-of-the-art classifiers to fail. Adversarial training, the standard defense, involves training on these difficult examples. However, the ideal objective—minimizing adversarial 0/1 loss—is mathematically intractable due to its non-smooth nature. Practitioners typically rely on surrogate losses like cross-entropy or the popular TRADES loss. Unfortunately, many of these surrogates lack **consistency**: there is no guarantee that minimizing the surrogate actually leads to the best possible robust model. This often results in a "robustness trade-off," where making a model robust significantly degrades its accuracy on normal, clean data. ## Methods: Principled Smooth Adversarial Loss (PSAL) The authors bridge the gap between theory and practice through three main contributions: 1. **Smooth Adversarial Losses:** They derive a new family of loss functions from classic multi-class losses (Max, Sum, and Constrained losses). These new functions decompose into a standard non-adversarial term and a "smooth" adversarial term based on the Lipschitz properties of the underlying functions. 2. **$$-consistency Guarantees:** Unlike standard consistency, $$-consistency is a stronger, more practical guarantee. It ensures that for a specific set of models (like neural networks), minimizing the surrogate loss directly minimizes the true adversarial error. 3. **The PSAL Algorithm:** The authors propose the **Principled Smooth Adversarial Loss (PSAL)** algorithm. It minimizes these theoretically grounded smooth losses using a regularized objective, employing standard Projected Gradient Descent (PGD) to handle the "inner maximization" of finding worst-case perturbations. ![Definitions of the new family of smooth adversarial losses (Max, Sum, and Constrained) derived from standard multi-class losses.](/figures/art_af8a583439f44a2ca410d7f2e06c9915) *[Source, p. 4](/pdf/art_dd859f46b9754ed396e05cdf59419f6e#page=4)* ## Key Insights: Why Consistency Matters A major finding of this research is the formal proof that **TRADES**, the long-standing gold standard for adversarial training, is not $$-consistent. Even in simple linear cases, TRADES can fail to find the most robust model, which explains why it often struggles to maintain high clean accuracy.

This paper suggests that the "robustness vs. accuracy" trade-off might partially be an artifact of using inconsistent loss functions. By using a consistent surrogate like PSAL, the authors achieved better robust accuracy while also significantly boosting clean accuracy.

Experimental Results

The PSAL algorithm was evaluated on CIFAR-10, CIFAR-100, and SVHN using WideResNet (WRN) architectures. It consistently outperformed TRADES and other state-of-the-art methods across all tested datasets.

Experimental results comparing PSAL against TRADES and other baselines. PSAL shows superior performance in both clean and robust (AutoAttack) accuracy across multiple datasets.
  • CIFAR-10 (WRN-70-16): PSAL achieved 86.63% clean accuracy (vs. 85.34% for TRADES) and 57.46% robust accuracy under the rigorous AutoAttack benchmark (vs. 57.05% for TRADES).
  • CIFAR-100: PSAL showed a significant jump in clean accuracy to 62.25% (compared to 60.56% for TRADES) while also improving robust accuracy.
  • SVHN: PSAL reached 94.31% clean accuracy, a notable improvement over the 93.03% achieved by TRADES.

Discussion and Limitations

While PSAL provides a "free lunch" in some scenarios—improving robustness without the typical heavy penalty on standard performance—it shares the high computational cost of PGD-based adversarial training. Additionally, the algorithm introduces new hyperparameters ($$ and $$) that require careful tuning. Future work could involve combining PSAL with "fast" adversarial training techniques to reduce overhead while maintaining the theoretical consistency benefits.

Summary updated

Abstract

Adversarial robustness is a critical property of classifiers in applications as they are increasingly deployed in complex real-world systems. Yet, achieving accurate adversarial robustness in machi...
BibTeX
@inproceedings{awasthi2023theoretically,
  author = {Pranjal Awasthi and Anqi Mao and Mehryar Mohri and Yutao Zhong},
  title = {{Theoretically Grounded Loss Functions and Algorithms for Adversarial Robustness}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=Q8Dyl-4lB7},
}
Figure