Polarizing Front Ends for Robust Cnns

Can Bakiskan, Soorya Gopalakrishnan, Metehan Cekic, Upamanyu Madhow, Ramtin PedarsaniICASSP 2020 · DOI

TL;DR

This paper introduces a novel defense mechanism for Convolutional Neural Networks (CNNs) against adversarial attacks. By employing a "polarizing" front-end that preprocesses input data to create well-separated clusters of features before quantization, the authors demonstrate a significant improvement in robustness, effectively eliminating perturbations under ideal conditions.

What Problem Does This Paper Address?

Deep neural networks, despite their impressive performance, are notoriously vulnerable to adversarial attacks. These attacks involve making tiny, often imperceptible, perturbations to input data (like images) that can cause the network to misclassify them with high confidence. While adversarial training has been a common defense strategy, it often lacks theoretical guarantees and can be computationally expensive. This paper tackles the problem of making CNNs more robust by focusing on a "bottom-up" approach: strengthening the network's defenses at the input layer before the data even reaches the core network.

Methods

The core of the proposed defense is a non-linear "polarizing" front-end. This front-end consists of:

  • Linear Transformation (Convolutional Filters): The input data is first transformed by a set of convolutional filters.
  • Polarization: The outputs of these filters are then normalized and processed to create distinct clusters of activations. The goal is to push these activations towards specific, well-separated values.
  • Quantization: Finally, these polarized activations are quantized using thresholds. The key insight is that if the clusters are sufficiently separated (by at least twice the perturbation budget), and the quantization thresholds are appropriately set, small perturbations cannot cause the activation to jump between quantization levels, thus eliminating their effect.

The authors propose a three-stage training procedure to learn these polarizing filters:

  1. Initialization and initial polarization: Filters are initialized randomly, and a "bump regularizer" is used to drive normalized activations away from zero towards the extremes (-1 and 1).
  2. Further polarization: A second bump regularizer is introduced to create three distinct clusters centered at -1, 0, and 1.
  3. Quantization and classifier training: A quantized non-linearity is introduced, and the front-end filters are frozen while the classifier is trained to adapt to the quantized outputs.

Datasets

The proposed defense was evaluated on the following datasets:

  • MNIST: A dataset of handwritten digits.
  • Fashion MNIST: A dataset of clothing items.

Key Insights and Ideas

  • "Excessive linearity" as a vulnerability: The paper posits that the vulnerability of deep neural networks to adversarial perturbations stems from their "excessive linearity."
  • Polarization as a path to robustness: The central idea is that by polarizing the input data into well-separated clusters, it becomes possible to eliminate adversarial perturbations through quantization.
  • Theoretical guarantees: Unlike empirical defenses like adversarial training, this polarization approach offers theoretical guarantees of robustness under ideal conditions.
  • Pixel bases are effective: The learned filters in the front-end resemble pixel bases, which is consistent with observations from adversarial training, suggesting a common underlying principle for robustness.
  • Bottom-up defense: The paper advocates for a bottom-up strategy, strengthening the network's input processing rather than solely relying on adversarial training of the entire network.

Key Results

  • The proposed defense significantly improves adversarial accuracy against FGSM, BIM, and PGD attacks on both MNIST and Fashion MNIST datasets.
  • For Fashion MNIST with an attack budget $\epsilon = 0.1$, the defense achieved 69.4% accuracy against FGSM, compared to 19.7% for no defense.
  • For MNIST with an attack budget $\epsilon = 0.3$, the defense achieved 93.2% accuracy against FGSM, compared to 21.9% for no defense.
  • While the defense improves robustness, it does not reach the accuracy levels of adversarial training, likely due to the practical impossibility of achieving perfect polarization.

Limitations and Future Work

  • Imperfect Polarization: The authors acknowledge that perfect polarization is not achievable in practice, leading to some leakage of adversarial perturbations.
  • Dataset Specificity: The three-stage training procedure was found to be sufficient for MNIST and Fashion MNIST, but may need to be repeated or adapted for more complex datasets.
  • Future Work:
    • Combining polarizing front-ends with non-linearities within the network to further attenuate attacks.
  • Developing polarizing bases for more complex datasets like CIFAR and ImageNet.
  • Exploring unsupervised learning of polarizing bases.

Summary updated

BibTeX
@inproceedings{bakiskan2020polarizing,
  author = {Can Bakiskan and Soorya Gopalakrishnan and Metehan Cekic and Upamanyu Madhow and Ramtin Pedarsani},
  title = {{Polarizing Front Ends for Robust Cnns}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053020},
}
Figure