Abstract
In order to train networks for verified adversarial robustness, previous work typically over-approximates the worst-case loss over (subsets of) perturbation regions or induces verifiability on top of adversarial training. The key to state-of-the-art performance lies in the expressivity of the employed loss function, which should be able to match the tightness of the verifiers to be employed post-training. We formalize a definition of expressivity, and show that it can be satisfied via simple convex combinations between adversarial attacks and IBP bounds. We then show that the resulting algorithms, named CC-IBP and MTL-IBP, yield state-of-the-art results across a variety of settings in spite of their conceptual simplicity. In particular, for $\ell_\infty$ perturbations of radius $\frac{1}{255}$ on TinyImageNet and downscaled ImageNet, MTL-IBP improves on the best standard and verified accuracies from the literature by from $1.98\%$ to $3.92\%$ points while only relying on single-step adversarial attacks.
Introduction
Deep neural networks are notoriously vulnerable to adversarial attacks—small, imperceptible perturbations that can flip a model's prediction. While empirical adversarial training provides some defense, it lacks formal guarantees. To achieve these guarantees, researchers employ verified training, which optimizes an upper bound on the worst-case loss over a perturbation region, often utilizing Interval Bound Propagation (IBP).
However, verified training is plagued by a significant "robustness-accuracy trade-off": models with high verified robustness often suffer from poor standard accuracy. While recent state-of-the-art methods like SABR have attempted to bridge this gap with complex heuristics, this paper investigates the fundamental property that makes these losses effective, identifying expressivity as the key ingredient.
The Expressivity Hypothesis
The authors hypothesize that the "secret sauce" of successful verified training is the ability of a loss function to smoothly interpolate between empirical adversarial loss and verified upper-bound loss. They define an expressive loss family $L_\alpha$ as one that satisfies three core criteria:
- Boundary Matching (Empirical): Matches the standard adversarial loss when $\alpha=0$.
- Boundary Matching (Verified): Matches the verified (IBP) loss when $\alpha=1$.
- Continuity and Monotonicity: Remains continuous and monotonically increasing with respect to the over-approximation coefficient $\alpha \in [0, 1]$.
This framework suggests that the specific form of the loss matters less than its ability to span the range between empirical and verified robustness.
Proposed Expressive Losses
To test this hypothesis, the authors propose three minimalistic expressive losses based on convex combinations:
- CC-IBP (Convex Combination IBP): Combines adversarial and over-approximated network outputs (logit differences) inside the loss function.
- MTL-IBP (Multi-Task Learning IBP): A simple weighted average of the adversarial loss and the verified loss.
- Exp-IBP (Exponential IBP): A convex combination of the logarithms of the losses, resulting in a product of losses raised to powers.
Crucially, the authors demonstrate that complex methods like SABR are members of this expressive family, implying their success is due to this property rather than specific geometric heuristics.
Experimental Evaluation
The proposed methods were evaluated on standard benchmarks: MNIST, CIFAR-10, TinyImageNet, and ImageNet64.
Navigating the Accuracy-Robustness Trade-off
The parameter $\alpha$ acts as a powerful knob for navigating the trade-off between standard and verified accuracy. As $\alpha$ increases, standard accuracy typically declines while verified accuracy (measured by verifiers like IBP, CROWN, or BaB) improves.
The optimal $\alpha$ varies significantly depending on the dataset and the perturbation radius $\epsilon$. For example, at $\epsilon=8/255$ on CIFAR-10, the sensitivity profile for MTL-IBP shifts.
State-of-the-Art Results
Simple expressive losses achieved state-of-the-art (SOTA) results, particularly on larger datasets where the gap between empirical and verified robustness is most pronounced:
- TinyImageNet ($\epsilon=1/255$): Exp-IBP achieved 26.18% verified robust accuracy, surpassing the previous best of 22.16% (STAPS).
- ImageNet64 ($\epsilon=1/255$): Exp-IBP reached 13.30% verified robust accuracy, a significant jump from the previous SOTA of 9.54% (SortNet).
- CIFAR-10 ($\epsilon=2/255$): CC-IBP achieved 63.78% verified accuracy, outperforming more complex methods like SABR and STAPS.
Discussion: The "Better Approximation" Fallacy
A common assumption in the field is that a tighter approximation of the true worst-case loss (e.g., using CROWN instead of IBP during training) will lead to better models. The authors show this is not necessarily true. Sometimes, a looser over-approximation provides a better training signal, leading to superior final performance. The smoothness and range provided by an expressive loss are often more critical than the tightness of the bound used during optimization.
Conclusion and Future Work
This work demonstrates that expressivity is a fundamental property for training provably robust networks. Simple convex combinations can outperform highly engineered methods by providing a flexible training signal. Future research directions include addressing the tuning overhead for $\alpha$, exploring non-ReLU architectures, and extending these concepts to probabilistic verification techniques like randomized smoothing.
Summary updated