Abstract
Deep neural networks possess a remarkable ability to generalize from training data to unseen test sets, despite having the capacity to memorize random labels. This study investigates the limits of this generalization by identifying the minimal training data corruptions required to induce a total collapse in test performance. By employing a novel evolutionary strategy, Maximum Domain Divergence based Evolutionary Strategy (MDD-ES), we demonstrate that modifying as few as five pixels per training image can cause a model to achieve near-perfect training accuracy while failing completely on clean test data. Our findings highlight a significant disparity in robustness between optimizers, revealing that while adaptive methods like ADAM are highly susceptible to these perturbations, vanilla Stochastic Gradient Descent (SGD) remains notably resilient. Furthermore, we challenge the prevailing "flat minima" hypothesis by showing that sharper minima can, in certain adversarial contexts, lead to superior generalization.
Introduction
The success of deep learning is often attributed to the ability of over-parametrized models to find solutions that generalize well. However, the mechanisms underlying this phenomenon remain poorly understood. While it is well-documented that neural networks can easily fit datasets with randomized labels, this research explores a more surgical intervention: Adversarial Distribution Shifts (ADS). Unlike random noise, ADS involves tiny, class-specific perturbations designed to mislead the model into learning "spurious features"—specifically the added noise—rather than the actual semantic content of the images.
Methodology: MDD-ES
To find the most damaging perturbations, the authors introduce the Maximum Domain Divergence based Evolutionary Strategy (MDD-ES). This approach utilizes the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a powerful black-box optimization algorithm, to search for optimal pixel locations and color values.
The search is guided by a specialized fitness function derived from domain adaptation theory, which balances two primary metrics:
- Semantic Mismatch: This measures the discrepancy in loss between clean images and their corrupted counterparts, ensuring the perturbation effectively shifts the model's focus.
- Domain Divergence: This metric quantifies the statistical distance between the corrupted training distribution and the clean test distribution.
By optimizing these factors, the MDD-ES identifies "vulnerable" pixels that, when modified, force the network to prioritize the background noise over the object of interest. The attack is remarkably sparse, targeting only a tiny fraction of pixels ($N_p \in \{1, 2, 5, 10\}$) per image. For each candidate perturbation, a CNN is trained from scratch to evaluate the resulting degradation in generalization.
The Fragility of Generalization
The experimental results across benchmarks such as MNIST, Fashion-MNIST, SVHN, and CIFAR-10 reveal that generalization is surprisingly fragile. On the SVHN dataset, a ResNet-32 model that typically achieves 92.8% accuracy on clean data sees its performance plummet to just 11.0% when trained on images with only five pixels modified via MDD-ES.
GradCAM visualizations confirm the mechanism of this failure. In models trained on corrupted data, the "attention" of the network shifts away from the central object and toward the specific coordinates of the perturbed pixels. The model effectively learns to use these tiny, invisible markers as shortcuts for classification, rendering it useless for real-world application.
Optimizer Robustness and the Flat Minima Paradox
A pivotal discovery of this research is the differing behavior of optimizers under adversarial pressure. Stochastic Gradient Descent (SGD) consistently outperformed adaptive optimizers like ADAM and RMSProp in terms of robustness. Adaptive optimizers appear to "tune" themselves into the geometry of the noise, more readily incorporating the spurious features into the model's decision boundaries.
This observation leads to a contradiction of the common "flat minima" theory, which posits that solutions located in flatter regions of the loss landscape generalize better. In this study, SGD often converged to sharper minima than ADAM, yet it maintained significantly higher test accuracy under corrupted training conditions. This suggests that the relationship between loss surface geometry and generalization is more complex than previously thought, particularly when the training data contains intentional, low-magnitude corruptions.
Conclusion
This work demonstrates that the generalization capabilities of deep neural networks can be dismantled with minimal effort through optimally evolved training perturbations. The resilience of SGD compared to adaptive optimizers suggests that the choice of optimization algorithm plays a crucial role in how models navigate noisy or adversarial data regimes. While the computational cost of searching for these perturbations remains high, the findings provide a new lens through which to view the relationship between loss surface geometry and model robustness.
Summary updated