Standard adversarial training is a specialist's game. When a model is trained to withstand $\ell_\infty$ perturbations (small changes to every pixel), it often becomes even more vulnerable to $\ell_1$ attacks (large changes to a few pixels). This "specialization gap" occurs because different mathematical norms represent fundamentally different geometric constraints, and optimizing for one typically ignores or compromises the others.
Recent research has shifted toward multi-perturbation robustness, aiming to create "generalist" models that maintain high accuracy across a union of threat models, such as $\ell_1, \ell_2$, and $\ell_\infty$ simultaneously.
The Specialization Problem
The core challenge, as noted by Changyou Chen and colleagues in On Norm-Agnostic Robustness of Adversarial Training, is that robustness is not a holistic property. Models optimized for a single norm often experience a total loss of accuracy when the attack norm changes. This suggests that the decision boundaries learned to defend against one type of noise do not naturally generalize to the geometry of others.
Finding the Robust Path
One intuitive solution is to find a "sweet spot" in the model's parameter space that satisfies multiple constraints. Can Chen and Pin Yu Chen propose Robust Mode Connectivity (RMC), which demonstrates that specialized models (e.g., one for $\ell_1$ and one for $\ell_\infty$) can be connected by a continuous path of high-performing, multi-robust parameters. Similarly, Ren Wang introduces Efficient Robust Mode Connectivity (EMRC) to identify these paths more effectively, allowing a single model to inherit the strengths of multiple specialists.
Architectural and Meta-Learning Strategies
Rather than forcing a single set of weights to do everything, some researchers use architectural modularity:
- Gated Batch Normalization: Aishan Liu and Xianglong Liu developed GBN, which treats different attack types as distinct "domains." The model uses gated branches to process different perturbations, neutralizing them before they reach the core layers.
- Hypernetworks: Huihui Gong and Chang Xu use Parameter-Saving Adversarial Training (PSAT), where a hypernetwork generates specialized "virtual" weights for different norms on the fly, significantly reducing the parameter overhead of maintaining multiple models.
- Meta-Learning: The Generalist++ framework by Yisen Wang uses a divide-and-conquer approach where specialized base learners share knowledge to mitigate the trade-off between clean accuracy and multi-norm robustness.
Provable and Game-Theoretic Guarantees
To move beyond empirical success, Enyi Jiang and Gagandeep Singh introduced CURE, a framework for certified robustness. By aligning the mathematical bounds of different $\ell_p$ norms during training, they provide deterministic guarantees that a model will remain secure against any attack within the union of specified perturbation sets.
Alternatively, Pranjal Awasthi and Da-Cheng Juan frame the problem as a game. Using the Multiplicative Weights Update (MWU) method, they train models to be robust across both different norms and different representation spaces (like pixels and frequency domains), ensuring the model doesn't over-index on defending against a single "player" or attack type.