Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses

Xiao Wang, Siyue Wang, Pin-Yu Chen, Yanzhi Wang, Brian Kulis, Xue Lin, Sang Chin

Abstract

Deep Neural Networks (DNNs) are vulnerable to adversarial examples, where small, intentional perturbations lead to incorrect classifications. While stochastic defenses introduce randomness to mitigate these attacks, they often suffer from a significant robustness-accuracy trade-off. This paper introduces Hierarchical Random Switching (HRS), a defense mechanism that switches between parallel, fully functional sub-channels at runtime. We evaluate HRS using a new metric, the Defense Efficiency Score (DES), demonstrating that it achieves superior protection with minimal impact on clean-data accuracy compared to traditional methods like Gaussian noise or defensive dropout.

Introduction

The fragility of DNNs against adversarial attacks remains a critical challenge. Stochastic defenses aim to confuse attackers by injecting randomness into the model's inference process. However, a persistent hurdle is the trade-off between robustness and accuracy: increasing "noise" to thwart attackers typically degrades performance on legitimate data. Current methods, such as adding Gaussian noise or using defensive dropout, often result in unacceptable accuracy drops. This research proposes Hierarchical Random Switching (HRS) to provide robust protection while maintaining high performance.

Methodology

The authors introduce two primary innovations: a standardized evaluation metric and a novel architectural defense.

Defense Efficiency Score (DES)

To quantify the trade-off between robustness and accuracy, the authors propose the Defense Efficiency Score: $$DES = \Delta d / \Delta t$$ where $\Delta d$ is the gain in defense rate and $\Delta t$ is the drop in test accuracy. A higher DES indicates a more efficient defense that provides greater protection per unit of accuracy lost.

Hierarchical Random Switching (HRS)

HRS restructures a network into several blocks, each containing multiple parallel channels with distinct weights.

  • Runtime Switching: For each input, a switcher randomly activates one channel per block, creating a unique "active path."
  • Decentralized Randomization: Unlike dropout, which disables components of a single model, every path in an HRS network is a fully functional, trained sub-network. This prevents attackers from finding a common weakness across all switching combinations.
  • Bottom-Up Training: A specialized training algorithm fixes weights block-by-block, ensuring subsequent layers adapt to the variations introduced by previous switching blocks.

Analysis of Robustness

A key finding is that defenses with higher gradient variance are more resilient to attacks. HRS naturally produces high gradient variance because the active path changes for every input, making it difficult for an attacker to craft a single effective perturbation.

Gradient distribution for HRS 30x30 showing high variance.

In contrast, traditional stochastic defenses like low-level dropout exhibit much lower variance, making them easier to bypass with standard adversarial attacks.

Gradient distribution for Dropout 0.1 showing significantly lower variance.

The authors also verified that HRS does not rely on "obfuscated gradients." Testing against adaptive attacks like Expectation Over Transformation (EOT) confirmed that the defense remains structurally sound.

Experimental Results

Evaluation on MNIST and CIFAR-10 benchmarks against FGSM, PGD, and CW attacks yielded the following insights:

  • Superior Trade-off: On CIFAR-10, HRS achieved a mean DES of 35.4, significantly outperforming adversarial training (~7.4) and Gaussian noise (~6.2).
  • Attack Resilience: Under a heavy PGD attack (8/255), HRS maintained its defense rate with only a 0.48% drop in test accuracy, whereas other defenses either failed completely or required much larger accuracy sacrifices.
  • Adversarial Reprogramming: HRS is the first defense demonstrated to be effective against adversarial reprogramming, where models are hijacked for unintended tasks.

Conclusion

Hierarchical Random Switching offers a promising path toward robust DNNs by leveraging decentralized randomness. While it increases memory requirements due to parallel channels and involves a more complex training process, its efficiency in the robustness-accuracy trade-off is unparalleled. Future work will focus on scaling HRS to larger architectures and datasets like ImageNet.

Summary updated

Abstract

Despite achieving remarkable success in various domains, recent studies have uncovered the vulnerability of deep neural networks to adversarial perturbations, creating concerns on model generalizability and new threats such as prediction-evasive misclassification or stealthy reprogramming. Among different defense proposals, stochastic network defenses such as random neuron activation pruning or random perturbation to layer inputs are shown to be promising for attack mitigation. However, one critical drawback of current defenses is that the robustness enhancement is at the cost of noticeable performance degradation on legitimate data, e.g., large drop in test accuracy. This paper is motivated by pursuing for a better trade-off between adversarial robustness and test accuracy for stochastic network defenses. We propose Defense Efficiency Score (DES), a comprehensive metric that measures the gain in unsuccessful attack attempts at the cost of drop in test accuracy of any defense. To achieve a better DES, we propose hierarchical random switching (HRS), which protects neural networks through a novel randomization scheme. A HRS-protected model contains several blocks of randomly switching channels to prevent adversaries from exploiting fixed model structures and parameters for their malicious purposes. Extensive experiments show that HRS is superior in defending against state-of-the-art white-box and adaptive adversarial misclassification attacks. We also demonstrate the effectiveness of HRS in defending adversarial reprogramming, which is the first defense against adversarial programs. Moreover, in most settings the average DES of HRS is at least 5X higher than current stochastic network defenses, validating its significantly improved robustness-accuracy trade-off.
BibTeX
@inproceedings{wang2019protecting,
  author = {Xiao Wang and Siyue Wang and Pin-Yu Chen and Yanzhi Wang and Brian Kulis and Xue Lin and Sang Chin},
  title = {{Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/833},
}
Figure