Robustness of Saak Transform Against Adversarial Attacks

Thiyagarajan Ramanathan, Abinaya Manimaran, Suya You, C.-c. Jay Kuo

TL;DR

This paper introduces a novel image classification system leveraging the multi-stage Saak transform to enhance robustness against adversarial attacks. By carefully selecting spectral dimensions at each stage, the system effectively denoises adversarial perturbations, leading to significantly improved classification accuracy compared to state-of-the-art defense mechanisms.

What Problem Does This Paper Address?

Deep learning models for image classification are notoriously vulnerable to adversarial attacks. These attacks involve subtly perturbing input images with imperceptible noise, causing models to misclassify them. This vulnerability poses a significant risk in safety-critical applications. While various defense mechanisms exist, they often fail to generalize to unseen attacks or require extensive retraining. This paper aims to develop a robust image classification system that inherently resists adversarial perturbations without relying on complex, attack-specific defenses.

Methods

The core of this work is a multi-stage Saak transform-based image classification pipeline. The Saak transform, inspired by deep learning, is a mathematically interpretable transform that decomposes images into spectral components. The authors extend this by using a multi-stage approach, where each stage applies the Saak transform, effectively creating a cascade of feature representations.

Key methodological contributions include:

  • Multi-stage Saak Transform: Applying the Saak transform iteratively to progressively extract features.
  • Cross-Entropy Based Feature Selection: At each stage, spectral dimensions are analyzed using cross-entropy to identify those that best discriminate between classes. This process is framed as an automatic denoising step, where dimensions with higher entropy (indicating noise or less discriminative power) are suppressed.
  • Transparent Feature Extraction and Representation: The entire process, from feature extraction to selection and classification, is designed to be transparent and computationally efficient, avoiding the need for backpropagation during feature extraction.
  • Kernel Extraction from Clean Data: Saak kernels are extracted using clean training images, and these kernels are then used to process both clean and adversarial images.

Datasets

The proposed method was evaluated on the following benchmark datasets:

  • MNIST
  • CIFAR-10
  • STL-10

Key Insights and Ideas

  • Spectral Dimension Discriminability: Adversarial and clean images exhibit different distributions in the Saak transform domain, particularly at higher spectral dimensions. This difference is key to detecting and mitigating adversarial perturbations.
  • Saak Transform as Automatic Denoising: The selection of spectral dimensions based on cross-entropy acts as an implicit denoising mechanism, effectively filtering out adversarial noise.
  • Inherent Robustness: The Saak transform's inherent properties, when integrated into a carefully designed pipeline, provide a robust defense against adversarial attacks without requiring explicit adversarial training or complex pre-processing modules.
  • Efficiency and Interpretability: The Saak transform's interpretability and the pipeline's low complexity make it an attractive alternative to more opaque deep learning defenses.

Key Results

The paper demonstrates that the proposed Saak transform-based classification system significantly outperforms state-of-the-art defense mechanisms across various adversarial attacks (FGSM, BIM, DeepFool) and datasets (MNIST, CIFAR-10, STL-10). The key metric reported is the drop in classification accuracy from clean images to attacked images.

  • MNIST: The Saak transform approach showed the lowest accuracy drop compared to all other defenses for all three attacks.
  • CIFAR-10 and STL-10: Even on more complex datasets where attacks are generally more effective, the Saak transform features exhibited a much lower accuracy drop than most other defenses. For example, on CIFAR-10 against FGSM, the Saak transform achieved a drop of only 25.1%, while other methods like JPEG compression or Bit Depth Reduction had drops exceeding 70%.
  • The paper also notes that the clean accuracy of the Saak transform system is competitive, with 99.4% on MNIST and 74.6% on CIFAR-10 (compared to VGG-16's 93.95% clean accuracy).

Limitations and Future Work

  • Generalization to Unseen Attacks: While the paper shows strong performance against common attacks, the authors do not explicitly discuss the system's robustness against novel or more sophisticated adversarial attack strategies.
  • Computational Cost of Multi-stage Transform: While the feature extraction is efficient, the multi-stage nature of the Saak transform might introduce some computational overhead compared to a single-stage classifier, though it's presented as less complex than adversarial training.
  • Future Work: The authors suggest that further exploration of feature selection strategies and the integration of Saak transform with other deep learning architectures could lead to even greater robustness. The inherent interpretability of the Saak transform also opens avenues for deeper analysis of adversarial phenomena.

Summary updated

Abstract

Image classification is vulnerable to adversarial attacks. This work investigates the robustness of Saak transform against adversarial attacks towards high performance image classification. We develop a complete image classification system based on multi-stage Saak transform. In the Saak transform domain, clean and adversarial images demonstrate different distributions at different spectral dimensions. Selection of the spectral dimensions at every stage can be viewed as an automatic denoising process. Motivated by this observation, we carefully design strategies of feature extraction, representation and classification that increase adversarial robustness. The performances with well-known datasets and attacks are demonstrated by extensive experimental evaluations.
BibTeX
@inproceedings{ramanathan2019robustness,
  author = {Thiyagarajan Ramanathan and Abinaya Manimaran and Suya You and C.-C. Jay Kuo},
  title = {{Robustness of Saak Transform Against Adversarial Attacks}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803240},
}
Figure