Abstract
Deep Neural Networks (DNNs) exhibit significant adversarial vulnerability, often varying drastically across different data classes and architectures. This paper presents Adversarial Double Machine Learning (ADML), a framework that reformulates adversarial robustness as a causal inference problem. By estimating the causal effect of perturbations on model predictions, ADML adaptively reweights training samples to prioritize the most vulnerable instances. Experimental results across multiple benchmarks and architectures (CNNs and Transformers) demonstrate that ADML effectively closes the robustness gap and outperforms state-of-the-art defense methods.
Introduction: The Robustness Gap
Despite the success of Adversarial Training (AT) as a primary defense against adversarial attacks, a persistent "robustness gap" remains. Standard models often show high robustness for certain classes while remaining highly susceptible in others. This non-uniformity suggests that current defenses, which typically treat all perturbations with equal weight, fail to address the underlying causal reasons why specific inputs are more fragile. This gap persists regardless of whether one utilizes deep CNNs or modern Vision Transformers.
Methodology: Adversarial Double Machine Learning (ADML)
ADML shifts the perspective from simple correlation to causal estimation. The framework identifies three key components: the original clean image (covariates, $x$), the adversarial perturbation (treatment, $t$), and the model's prediction (outcome, $y$). The goal is to estimate a causal parameter $\theta$ that measures the direct impact of the perturbation on the prediction.
To estimate $\theta$ accurately without falling into statistical traps, ADML employs Double Machine Learning (DML) techniques, including sample-splitting and cross-fitting. A "balancing ratio" $\tau$ is then calculated based on the estimated causal parameter. This ratio acts as a dynamic weight during training: samples with high causal vulnerability (high $\theta$) receive greater emphasis in the loss function, forcing the model to learn more robust features for its weakest points. To maintain efficiency, the authors utilize a gradient approximation that combines attacked confidence and input gradients into a computationally feasible metric.
Key Results and Discussion
The effectiveness of ADML was evaluated on CIFAR-10, CIFAR-100, and ImageNet using various backbones, including VGG, ResNet, and Vision Transformers (ViT/DeiT). ADML consistently outperformed standard defense baselines like TRADES, MART, and AWP.
As shown in the results, ADML significantly raises the "robustness floor." For instance, on CIFAR-10 with a VGG-16 backbone, robustness against Auto-Attack improved from 46.3% to 54.5%. Crucially, the improvement is most pronounced in the most vulnerable segments of the data, leading to more consistent performance across the entire dataset. The framework's success across both CNNs and Transformers indicates that the causal nature of adversarial vulnerability is a fundamental property shared across different neural architectures.
Conclusion
Adversarial Double Machine Learning provides a principled causal approach to enhancing model robustness. By identifying and focusing on the causal drivers of misclassification under attack, ADML offers a more nuanced and effective defense than traditional uniform adversarial training. While it introduces some computational overhead due to sample-splitting, the gains in robustness consistency make it a powerful tool for securing deep learning models. Future work may explore extending this causal framework to address other forms of distribution shift and natural corruptions.
Summary updated