Towards Accurate and Compact Architectures via Neural Architecture Transformer

Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Zhipeng Li, Jian Chen, Peilin Zhao, Junzhou Huang

TL;DR

Even state-of-the-art neural networks contain redundant operations that bloat computational cost and hinder performance. This paper introduces Neural Architecture Transformer (NAT) and its advanced version, NAT++, which automatically optimize any given architecture by replacing inefficient operations with lighter alternatives (like skip connections or separable convolutions) without increasing the overall compute budget. By framing this as a Markov Decision Process solved via a Graph Convolutional Network, the authors achieve higher accuracy and lower computational costs on both hand-crafted and NAS-generated models.

What Problem Does This Paper Address?

Designing neural network architectures is notoriously difficult. While human experts have crafted highly successful models (like ResNet and VGG) and Neural Architecture Search (NAS) algorithms have automated the discovery of others (like DARTS), the resulting architectures are rarely perfect. They often contain redundant or non-significant operations—such as unnecessary intermediate convolution or pooling layers.

This redundancy wastes memory and compute, and can even degrade predictive performance. Existing optimization methods, like Neural Architecture Optimization (NAO), attempt to fix this but often output entirely different architectures that are hard to analyze, or they improve accuracy at the expense of adding more parameters and computational overhead. The authors tackle the challenge of taking an existing architecture and strictly optimizing it to be more compact and accurate, without starting the search from scratch or increasing its computational cost.

Methods

The authors propose two iterative methods to transform architectures:

  • Neural Architecture Transformer (NAT): NAT treats architecture optimization as a Markov Decision Process (MDP). It uses a Graph Convolutional Network (GCN) to "look" at the architecture's graph structure and decide which operations to replace with either a "skip connection" or a "null connection" (removing the edge entirely).
  • NAT++: Recognizing that simply dropping operations limits the search space, the authors introduce NAT++. This version uses a two-level transition rule that allows for more granular downgrades. It can change the type of operation (e.g., standard convolution $\rightarrow$ separable convolution $\rightarrow$ pooling) and the kernel size (e.g., 5x5 $\rightarrow$ 3x3 $\rightarrow$ 1x1).
  • Binary-Masked Softmax (BMSoftmax): To guarantee that the optimized architecture never exceeds the original's computational cost, NAT++ uses a custom BMSoftmax layer. This layer acts as a hard filter, zeroing out the probabilities of any "invalid" transitions that would increase the compute budget.
  • Training: Because accuracy and computational cost are non-differentiable, the GCN controller is trained using Reinforcement Learning (Policy Gradient) to maximize the expected performance improvement on a validation set.

Datasets

The authors evaluate their methods across standard image classification and face recognition benchmarks:

  • Image Classification: CIFAR-10, CIFAR-100, and ImageNet.
  • Face Recognition: LFW, CFP-FP, and AgeDB-30.

Key Insights and Ideas

  • Strict Cost Constraints Work: By strictly forcing the optimizer to only choose transitions that reduce or maintain computational cost (via the BMSoftmax layer), the model naturally discovers more efficient sub-graphs without needing complex, hard-to-tune penalty terms in the reward function.
  • Granular Transitions Matter: NAT++ proves that simply dropping layers (as NAT does) leaves performance on the table. Allowing granular downgrades—like swapping a heavy 5x5 standard convolution for a lighter 3x3 separable convolution—provides a smoother optimization path and yields better architectures.
  • Mitigating NAS Bias: A known flaw in NAS is that networks with many skip connections train faster in early epochs, tricking the search algorithm into preferring them despite poor final generalization. NAT/NAT++ avoids this trap by decoupling the supernet training from the search policy. By using uniform sampling during training, all operations get an equal chance to train, preventing the algorithm from unfairly favoring fast-converging skip connections.

Key Results

The proposed methods consistently improved both hand-crafted and automatically searched architectures:

  • Hand-crafted models: On ImageNet, NAT++ boosted VGG16's Top-1 accuracy from 71.6% to 74.7% while reducing Multiply-Adds (MAdds) from 15.6G to 14.9G. ResNet50 improved from 76.2% to 77.8% with a slight reduction in compute.
  • NAS models: When applied to the NAS-generated DARTS model on ImageNet, NAT++ boosted Top-1 accuracy from 73.1% to 73.9% while slashing computational cost by roughly 28% (574M MAdds down to 413M MAdds).
  • Face Recognition: NAT++ successfully optimized LResNet34E-IR and MobileFaceNet, yielding state-of-the-art accuracy across LFW, CFP-FP, and AgeDB-30 without adding any computational overhead.

Limitations and Future Work

  • Diminishing Returns on Highly Optimized Models: The authors note that NAT and NAT++ yield identical results when optimizing MobileNetV2. Because MobileNetV2 is already built almost entirely of highly efficient 1x1 convolutions and 3x3 separable convolutions, NAT++'s extended transition rules have no "cheaper" valid operations to transition to.
  • GCN Depth Limits: The authors found that building a deeper GCN controller (e.g., 5 or 10 layers) actually degrades performance by introducing redundant information from multi-hop neighbors. They had to limit the GCN to 2 layers.
  • Future Work: While not explicitly mapped out in a dedicated section, the paper implies that future work could involve automatically learning the transition rules (rather than relying on the hard-coded two-level rules) to better optimize already-compact architectures like MobileNetV2.

Summary updated

Abstract

Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-designed/searched architecture may still contain many nonsignificant or redundant modules/operations (e.g., some intermediate convolution or pooling layers). Such redundancy may not only incur substantial memory consumption and computational cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computational cost. To this end, we have proposed a Neural Architecture Transformer (NAT) method which casts the optimization problem into a Markov Decision Process (MDP) and seeks to replace the redundant operations with more efficient operations, such as skip or null connection. Note that NAT only considers a small number of possible replacements/transitions and thus comes with a limited search space. As a result, such a small search space may hamper the performance of architecture optimization. To address this issue, we propose a Neural Architecture Transformer++ (NAT++) method which further enlarges the set of candidate transitions to improve the performance of architecture optimization. Specifically, we present a two-level transition rule to obtain valid transitions, i.e., allowing operations to have more efficient types (e.g., convolution <inline-formula><tex-math notation="LaTeX">${\to }$</tex-math></inline-formula> separable convolution) or smaller kernel sizes (e.g., <inline-formula><tex-math notation="LaTeX">$5{\times }5 {\to } 3{\times }3$</tex-math></inline-formula> ). Note that different operations may have different valid transitions. We further propose a Binary-Masked Softmax (BMSoftmax) layer to omit the possible invalid transitions. Last, based on the MDP formulation, we apply policy gradient to learn an optimal policy, which will be used to infer the optimized architectures. Extensive experiments show that the transformed architectures significantly outperform both their original counterparts and the architectures optimized by existing methods.
BibTeX
@article{guo2022towards,
  author = {Yong Guo and Yin Zheng and Mingkui Tan and Qi Chen and Zhipeng Li and Jian Chen and Peilin Zhao and Junzhou Huang},
  title = {{Towards Accurate and Compact Architectures via Neural Architecture Transformer}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3086914},
}
Figure