Understanding Generalization in Neural Networks for Robustness against Adversarial Vulnerabilities

Subhajit ChaudhuryAAAI 2020 · DOI

Introduction

Neural networks are notoriously fragile. A tiny, imperceptible change to an image can cause a state-of-the-art model to misclassify it entirely. These vulnerabilities typically manifest as adversarial attacks (manipulating data at test time) or poisoning attacks (manipulating data during training). This research frames adversarial vulnerability not just as a security flaw, but as a fundamental failure of neural networks to generalize in a human-like manner. The core hypothesis is that robustness is a form of generalization: a truly robust model learns the essence of an object rather than relying on "spurious correlations" or shortcuts that are easily manipulated.

The Single-Pixel Poisoning Attack

To investigate this, a novel "single-pixel" data poisoning attack was developed. Unlike complex perturbations, this attack injects a single malicious pixel into training images to divert the model's attention to non-salient regions—parts of the image typically ignored by the model.

Using GradCAM for visualization, the researchers identified these non-salient areas. They then employed CMA-ES, a gradient-free optimization technique, to find the specific pixel location and intensity that would most effectively trick the model. This method provides an interpretable way to test how easily a model's learning process can be derailed by minimal, high-frequency noise.

Model Architecture and Resilience

The study compared how different learning algorithms handle these poisons, ranging from traditional machine learning models to deep neural networks. A key finding was the remarkable resilience of non-linear models, specifically RBF-SVMs (Support Vector Machines with a Radial Basis Function kernel).

Because RBF-SVMs rely on Euclidean distance in a transformed feature space, a single-pixel change represents a negligible mathematical shift. This allows the model to effectively ignore the poison, whereas linear models like the Linear SVM are devastated by the same attack.

Comparison of testing error (%) across different machine learning models under attack.

The Optimizer Paradox: SGD vs. Adaptive Methods

Perhaps the most surprising insight involves the choice of optimization algorithm. While adaptive optimizers like Adam, RMSProp, and Adabound are popular for their fast convergence, they proved to be significantly more vulnerable to poisoning. These algorithms adjust learning rates for individual parameters, a "smart" behavior that appears to make them more prone to memorizing the malicious, spurious patterns introduced by the attacker.

In contrast, vanilla Stochastic Gradient Descent (SGD) showed significant resilience. Its simpler path to convergence acts as a natural filter against the high-frequency noise of a single-pixel poison. On datasets like SVHN, models trained with SGD maintained high accuracy even under attack, while those trained with adaptive methods saw their testing accuracy plummet toward zero.

Testing accuracy over epochs for various optimizers under single-pixel poisoning on MNIST and SVHN.

Conclusion

This research highlights that the quest for faster convergence through adaptive optimizers may come at the cost of robustness. By demonstrating that SGD and non-linear feature transformations like RBF kernels provide inherent protection against single-pixel poisons, the study suggests that future "robust-by-design" models should prioritize simpler optimization paths and more robust feature representations. While single-pixel attacks are a specific "unit test," they reveal fundamental differences in how models generalize, pointing the way toward more resilient artificial intelligence.

Summary updated

BibTeX
@inproceedings{chaudhury2020understanding,
  author = {Subhajit Chaudhury},
  title = {{Understanding Generalization in Neural Networks for Robustness against Adversarial Vulnerabilities}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7129},
}
Figure