Abstract
Deep neural networks (DNNs) are highly susceptible to adversarial attacks—imperceptible perturbations that lead to incorrect classifications. While adversarial training (AT) is currently the most effective defense, it typically introduces a trade-off: increasing robustness often degrades the model's generalization on natural data. This paper introduces the concept of the Confidence Threshold (CT) and proposes Confidence Threshold Reduction (CTR) as a strategy to improve both robustness and generalization simultaneously. We present two novel loss functions: the Mask-guided Divergence Loss (MDL) for natural training and the Standard Deviation Loss (STD) for adversarial training. Empirical and theoretical analyses demonstrate that these methods effectively mitigate the robustness-accuracy trade-off across multiple benchmarks.
Introduction
The vulnerability of DNNs to adversarial examples remains a significant hurdle for their deployment in safety-critical applications. The current gold standard for defense, Adversarial Training (AT), solves a min-max optimization problem that often results in a model that is robust but less accurate on clean, "natural" images. This tension between robustness and generalization is a well-documented challenge in the field.
This research proposes that this trade-off can be addressed by focusing on the model's Confidence Threshold (CT). By reducing the CT—the probability level at which a model's prediction becomes reliable—we can create models that are easier to satisfy for correct classifications and more difficult for adversarial perturbations to deceive.
The Concept of Confidence Threshold Reduction (CTR)
The Confidence Threshold ($\hat{p}$) is defined as the probability level above which a model is almost certainly correct. In a standard 10-class classifier, this threshold is typically high (e.g., 0.5). If the threshold can be lowered (e.g., to 0.11), the model becomes more resilient. CTR aims to achieve this by restructuring how the model handles class probabilities, particularly those of the incorrect categories.
Methodology
1. Natural Training: Mask-guided Divergence Loss (MDL)
For standard (natural) training, the authors propose the MDL function. This method leverages the diversity within a network by treating a model using Dropout as an ensemble of sub-networks.
- Diversity through Orthogonality: MDL introduces an orthogonal term to the loss function that minimizes the cosine similarity between the probability vectors of the incorrect categories across different sub-networks. This forces sub-networks to "disagree" on why a wrong answer is wrong, leading to a more stable and generalized global model.
- Masking Mechanism: To prevent the model from overfitting to noisy or extremely difficult samples, a masking mechanism is employed. This ensures the diversity push is only applied to samples where the model is already performing reasonably well.
2. Adversarial Training: Standard Deviation Loss (STD)
Because adversarial training frameworks rarely utilize Dropout, the authors developed the STD loss to reduce the CT in AT settings.
- Uniformity of Error: The STD loss minimizes the standard deviation of the probabilities assigned to all incorrect classes. By forcing the distribution of "wrong" probabilities to be uniform and low, the model naturally becomes more certain about the correct class at a lower absolute probability.
- Integration: This loss is easily integrated into existing adversarial frameworks like TRADES or Madry-AT as a modified cross-entropy or KL-divergence term.
Experimental Results
The proposed methods were evaluated on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 using architectures such as VGG16, ResNet18, and ResNet50.
Generalization and Robustness in Natural Training
On the CIFAR-100 dataset, MDL improved natural accuracy by up to 1.38% compared to standard Dropout. Remarkably, naturally trained models using MDL achieved robustness levels (e.g., ~62% accuracy under FGSM attack on CIFAR-10) that approached the performance of some dedicated adversarial training methods, without the computational cost of generating adversarial examples during training.
Enhancing Adversarial Training
When integrated into the TRADES framework, the CTR approach successfully broke the traditional trade-off. It improved natural accuracy by 1.27% to 1.64% while simultaneously increasing robust accuracy against a battery of attacks, including PGD and AutoAttack.
The "STD Attack"
An interesting byproduct of this research was the discovery of the STD Attack. The authors found that attacks designed to maximize the standard deviation of incorrect class probabilities (S-PGD, S-APGD) were more effective at breaking models than standard cross-entropy-based attacks. This suggests that the distribution of incorrect class probabilities is a critical, yet often overlooked, vulnerability in DNNs.
Conclusion
Confidence Threshold Reduction represents a promising shift in how we approach model training. By moving away from simply maximizing top-1 confidence and instead focusing on the distribution of incorrect class probabilities, we can develop models that are both more accurate and more secure. While MDL introduces some computational overhead due to multiple forward passes, the dual gains in generalization and robustness provide a compelling case for its adoption in robust deep learning pipelines.
Summary updated