Tiny Adversarial Multi-Objective Oneshot Neural Architecture Search

Guoyang Xie, Jinbao Wang, Guo Yu, Jiayi Lyu, Feng Zheng, Yaochu Jin

TL;DR

This paper introduces TAM-NAS, a novel one-shot neural architecture search (NAS) method designed to find tiny neural networks (TNNs) that are robust against adversarial attacks without sacrificing clean accuracy or significantly increasing model size. By optimizing for adversarial accuracy, clean accuracy, and model size simultaneously, TAM-NAS provides a practical approach to developing more secure and efficient TNNs for resource-constrained mobile devices.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are notoriously vulnerable to adversarial attacks, where small, imperceptible perturbations to input data can cause misclassifications. While larger models generally exhibit better robustness, this is often impractical for mobile devices, which require "tiny" neural networks (TNNs) due to limitations in computational power and energy consumption. Existing research has focused on improving robustness by increasing model size or employing complex adversarial training techniques, but there's a significant gap in methods that specifically enhance the robustness of TNNs without compromising their efficiency or accuracy. This paper aims to bridge this gap by developing a NAS framework that can discover TNN architectures offering a favorable trade-off between adversarial robustness, clean accuracy, and model size.

Methods

The core of this work is the Tiny Adversarial Multi-Objective Oneshot NAS (TAM-NAS) framework. Here's a breakdown of its key components:

  • Novel Search Space: The authors design a new search space tailored for TNNs, incorporating "tiny robust blocks" and channels. This space is built upon the ShuffleNetV2 architecture and includes variations of blocks like SE layers and non-local blocks, aiming to balance model size with adversarial performance.
  • One-Shot NAS: TAM-NAS employs a one-shot NAS approach. This involves training a single, large "supernet" that encompasses all possible subnet architectures within the defined search space. Subnets are then evaluated by inheriting weights from this supernet, significantly reducing the computational cost compared to training each architecture from scratch.
  • Multi-Objective Optimization: The NAS process is framed as a multi-objective optimization problem. The three objectives are:
  • Adversarial Accuracy: Performance against adversarial attacks (specifically, white-box PGD attacks).
  • Clean Accuracy: Performance on unperturbed data.
  • Model Size: The number of parameters in the network. The NSGA-II (Non-dominated Sorting Genetic Algorithm II) is used to find the Pareto optimal set of architectures that represent the best trade-offs among these objectives.

  • Adversarial Training Paradigm: The paper proposes a new adversarial training strategy for the supernet. They investigate how the supernet's adversarial training parameters (perturbation range, attack steps) influence the robustness of the sampled subnets. They also explore the impact of training subnets from scratch versus fine-tuning inherited weights.

  • Statistical Analysis for Design Guidelines: After identifying the best trade-off architectures, the authors perform a statistical analysis of the block and channel choices across layers. This analysis aims to provide practical guidelines for designing robust TNNs.

Datasets

The experiments were conducted on two standard datasets:

  • CIFAR-10
  • SVHN

Key Insights and Ideas

  • Supernet's Role: The supernet's primary role is to discover the optimal architecture, not necessarily the optimal weights. This is a crucial insight that explains why training subnets from scratch often outperforms fine-tuning inherited weights.
  • Training from Scratch Advantage: For the non-dominated subnets identified by the multi-objective search, training from scratch consistently yielded better adversarial and clean accuracy compared to fine-tuning weights from the supernet. This suggests that the supernet's learned weights might not be perfectly suited for all optimal architectures it discovers.
  • Layer-wise Design Principles: The analysis reveals a trend for robust TNN design:
  • Blocks: Robust blocks (pure robust or tiny robust) are beneficial in the initial layers to mitigate pixel-wise perturbations, while pure tiny blocks are preferred in later layers to balance performance and size.
  • Channels: Wider intermediate channels in early layers help with adversarial defense, while gradually decreasing channel width in later layers maintains efficiency.
  • Epsilon Size vs. Attack Steps: Adversarial accuracy is more sensitive to the epsilon size (perturbation magnitude) than to the number of attack steps during supernet training. This allows for more efficient supernet training by reducing attack steps without significant performance loss.

Key Results

  • The proposed TAM-NAS framework successfully identifies tiny neural networks that achieve a favorable trade-off between adversarial accuracy, clean accuracy, and model size.
  • The "Lego-Net" architecture, designed based on the derived block and channel statistics, demonstrated improved adversarial accuracy (e.g., 10.36% increase on CIFAR-10 for a specific attack) and reduced model size compared to state-of-the-art hand-crafted tiny networks like ShuffleNetV2 and MobileNetV3.
  • Training subnets from scratch after the NAS search significantly improved both adversarial and clean accuracy for the identified non-dominated architectures.
  • The paper provides concrete guidelines for designing robust TNNs, which were validated through experiments.

Limitations and Future Work

  • Supernet Dependency: The performance of the discovered subnets is still significantly influenced by the architecture of the supernet.
  • Future Work Suggestion: The authors propose a "co-evolutionary multi-objective NAS framework" where the supernet architecture itself could evolve during the search process, potentially leading to even more optimal solutions.

Summary updated

Abstract

The widely employed tiny neural networks (TNNs) in mobile devices are vulnerable to adversarial attacks. However, more advanced research on the robustness of TNNs is highly in demand. This work focuses on improving the robustness of TNNs without sacrificing the model's accuracy. To find the optimal trade-off networks in terms of the adversarial accuracy, clean accuracy, and model size, we present TAM-NAS, a tiny adversarial multi-objective one-shot network architecture search method. First, we build a novel search space comprised of new tiny blocks and channels to establish a balance between the model size and adversarial performance. Then, we demonstrate how the supernet facilitates the acquisition of the optimal subnet under white-box adversarial attacks, provided that the supernet significantly impacts the subnet's performance. Concretely, we investigate a new adversarial training paradigm by evaluating the adversarial transferability, the width of the supernet, and the distinction between training subnets from scratch and fine-tuning. Finally, we undertake statistical analysis for the layer-wise combination of specific blocks and channels on the first non-dominated front, which can be utilized as a design guideline for the design of TNNs.
BibTeX
@inproceedings{xie2023tiny,
  author = {Guoyang Xie and Jinbao Wang and GUO YU and Jiayi Lyu and Feng Zheng and Yaochu Jin},
  title = {{Tiny Adversarial Multi-Objective Oneshot Neural Architecture Search}},
  booktitle = {Complex},
  year = {2023},
  doi = {10.1007/s40747-023-01139-8},
}
Figure