Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses

Xiao Wang, Siyue Wang, Pin-Yu Chen, Yanzhi Wang, Brian Kulis, Xue Lin, Sang Peter Chin

Deep Neural Networks (DNNs) are notoriously fragile. Small, calculated perturbations to an input—known as adversarial attacks—can cause a model to confidently misclassify an image or even be "reprogrammed" to perform an entirely different task. To counter this, researchers have developed stochastic defenses (like random dropout or adding Gaussian noise) that introduce randomness to "dither" the model's gradients, making it harder for an attacker to find a successful perturbation. However, most current defenses significantly degrade the model's performance on legitimate, "clean" data.

This paper proposes Hierarchical Random Switching (HRS), a novel defense mechanism that breaks this cycle by creating a defense that is robust against sophisticated white-box attacks without sacrificing the model's primary utility.

Hierarchical Random Switching (HRS)

The core innovation is the HRS-protected model. Instead of a standard chain of layers, the network is divided into several "switching blocks." Each block contains multiple parallel channels with the same architecture but different weights.

Architecture of the HRS model compared to a base model.

Key Methodological Components:

  1. Random Switching: At runtime, for every input, a "switcher" randomly selects one active channel per block. The input propagates through a unique, randomly assembled path.
  2. Decentralized Randomization: Unlike an ensemble (which runs multiple full models) or dropout (which breaks parts of a single model), HRS ensures that every possible path is a fully functional, high-performing version of the base architecture.
  3. Bottom-Up Training: To ensure all paths work well together, the authors use a sequential training algorithm. They train the first block's channels to convergence, freeze them, and then train the next block's channels to adapt to the variations produced by the previous blocks.

Evaluation and the Defense Efficiency Score (DES)

To standardize comparisons across different defense methods, the authors define the Defense Efficiency Score (DES):

$$DES = \frac{\Delta d}{\Delta t}$$

where $\Delta d$ is the gain in defense rate and $\Delta t$ is the drop in test accuracy. This metric allows for a "bang-for-your-buck" assessment of how much robustness a defense provides relative to its cost in clean-data accuracy.

Key Results

The researchers evaluated HRS on MNIST and CIFAR-10 benchmarks against state-of-the-art attacks, including Projected Gradient Descent (PGD) and Expectation Over Transformation (EOT).

Defense Rate vs. Test Accuracy Drop for various defense methods.
  • Superior Trade-off: As shown in the figure above, HRS (blue triangles) maintains a significantly steeper slope than traditional methods. On CIFAR-10, HRS achieved a Mean DES of 35.4, compared to just 6.1 for Gaussian noise and 7.3 for Adversarial Training.
  • White-Box Resilience: Against PGD attacks, HRS maintained a high defense rate (~50-60%) with less than a 0.5% drop in test accuracy. Other defenses like SAP or Dropout were completely broken (0% defense) at similar accuracy levels.
  • First Defense Against Reprogramming: Adversarial reprogramming is a stealthy threat where an attacker uses an input wrapper to make an AI perform an unintended task. HRS is the first defense shown to effectively mitigate this threat.

Insights and Limitations

HRS succeeds because it is "constructive" rather than "destructive." Traditional stochastic defenses like SAP remove information to create randomness, whereas HRS switches between different, equally valid ways of processing information. This maintains high clean-data accuracy while introducing the gradient variance necessary to foil attackers.

However, HRS does introduce storage overhead. While only one path is active at runtime (keeping computational cost low), the model must store the weights for all parallel channels. Future work will focus on scaling HRS to massive architectures like Transformers and exploring parallelized training for the switching blocks.

Summary updated

Abstract

Despite achieving remarkable success in various domains, recent studies have uncovered the vulnerability of deep neural networks to adversarial perturbations, creating concerns on model generalizability and new threats such as prediction-evasive misclassification or stealthy reprogramming. Among different defense proposals, stochastic network defenses such as random neuron activation pruning or random perturbation to layer inputs are shown to be promising for attack mitigation. However, one critical drawback of current defenses is that the robustness enhancement is at the cost of noticeable performance degradation on legitimate data, e.g., large drop in test accuracy.This paper is motivated by pursuing for a better trade-off between adversarial robustness and test accuracy for stochastic network defenses. We propose Defense Efficiency Score (DES), a comprehensive metric that measures the gain in unsuccessful attack attempts at the cost of drop in test accuracy of any defense. To achieve a better DES, we propose hierarchical random switching (HRS), which protects neural networks through a novel randomization scheme. A HRS-protected model contains several blocks of randomly switching channels to prevent adversaries from exploiting fixed model structures and parameters for their malicious purposes. Extensive experiments show that HRS is superior in defending against state-of-the-art white-box and adaptive adversarial misclassification attacks. We also demonstrate the effectiveness of HRS in defending adversarial reprogramming, which is the first defense against adversarial programs. Moreover, in most settings the average DES of HRS is at least 5X higher than current stochastic network defenses, validating its significantly improved robustness-accuracy trade-off.
BibTeX
@inproceedings{wang2019protecting,
  author = {Xiao Wang and Siyue Wang and Pin-Yu Chen and Yanzhi Wang and Brian Kulis and Xue Lin and Sang Peter Chin},
  title = {{Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/833},
}
Figure