Towards Accurate and Compact Architectures via Neural Architecture Transformer

Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Zhipeng Li, Jian Chen, Peilin Zhao, Junzhou Huang

Introduction

Deep learning architectures, whether human-designed or automatically searched, often contain redundant operations that bloat computational cost and hinder performance. This paper introduces Neural Architecture Transformer (NAT) and its advanced version, NAT++, which use reinforcement learning and graph convolutional networks to automatically replace inefficient operations with lighter alternatives. The result is a plug-and-play optimizer that compresses existing models while simultaneously boosting their accuracy.

The Challenge of Architectural Redundancy

Designing neural networks is a complex task. While Neural Architecture Search (NAS) has largely automated this process, the resulting architectures—much like human-crafted ones such as VGG or ResNet—often contain "junk DNA." These are redundant or non-significant operations, such as unnecessary convolution or pooling layers, that consume memory and compute without improving predictive power.

Existing optimization methods, like Neural Architecture Optimization (NAO), attempt to address this but face significant drawbacks. They often alter the original architecture so drastically that it becomes difficult to analyze, or they inadvertently increase computational costs during optimization. NAT aims to be a universal, plug-and-play architecture optimizer that takes any given network and strictly improves performance while guaranteeing a reduced or maintained computational footprint.

Methodology: Neural Architecture Transformer (NAT)

The authors frame architecture optimization as a Markov Decision Process (MDP) solved via Reinforcement Learning, specifically using Policy Gradient.

NAT and NAT++

The base NAT method utilizes a Graph Convolutional Network (GCN) as a controller. It analyzes the network's graph structure and determines whether to retain an operation, replace it with a "skip connection," or remove it entirely via a "null connection."

The transition logic between original (O), skip (S), and null (N) connections.

To expand the search space, NAT++ introduces a two-level transition rule:

  • Type Transition: Downgrading compute-heavy operations to lighter alternatives (e.g., Standard Convolution $\rightarrow$ Separable Convolution $\rightarrow$ Pooling).
  • Kernel Transition: Reducing kernel sizes (e.g., 5x5 $\rightarrow$ 3x3 $\rightarrow$ 1x1).

Optimization Constraints

To ensure a strictly downward cost, the authors introduce Binary-Masked Softmax (BMSoftmax). This layer in the controller mathematically masks out any transitions that would increase computational cost (e.g., moving from a 3x3 to a 5x5 kernel), ensuring the optimized model fits within the original's resource constraints.

Furthermore, to mitigate "fast-convergence bias"—where networks with many skip connections appear better early in training—the authors use Decoupled Supernet Training. By uniformly sampling architectures during supernet training, NAT++ ensures all operation types are evaluated fairly based on final performance rather than initial training speed.

Key Results and Evaluation

NAT++ was evaluated across diverse benchmarks, including Image Classification (CIFAR, ImageNet) and Face Recognition (LFW, CFP-FP, AgeDB-30).

Comparison of architectures obtained by NAO and NAT against the original structure. NAT achieves higher accuracy with fewer parameters.

The results demonstrate consistent "free" improvements:

  • Hand-crafted Models: VGG16 on ImageNet improved from 71.6% to 74.7% Top-1 accuracy, while parameters decreased from 138M to 131M.
  • NAS Models: For the DARTS architecture on ImageNet, NAT++ boosted Top-1 accuracy from 73.1% to 73.9% and reduced parameters by approximately 19% (4.7M to 3.8M).
  • Face Recognition: LResNet34E-IR accuracy on LFW increased from 99.72% to 99.83% with reduced computation.

Discussion and Limitations

While highly effective, NAT++ shows diminishing returns on models that are already hyper-compact, such as MobileNetV2. Since MobileNetV2 already utilizes 1x1 and 3x3 separable convolutions, there are few valid "downward" transitions left for the optimizer to exploit. Additionally, the search process is sensitive to the entropy regularization weight ($\lambda$), requiring careful tuning to balance exploration.

Conclusion

Neural Architecture Transformer provides a robust framework for refining both manual and automated neural designs. By enforcing computational constraints through transition rules and leveraging graph-aware controllers, it offers a reliable path toward more efficient and accurate deep learning models. Future work may extend this approach to non-vision domains like NLP or integrate it as a continuous pruning mechanism within the NAS pipeline.

Summary updated

BibTeX
@article{guo2022towards,
  author = {Yong Guo and Yin Zheng and Mingkui Tan and Qi Chen and Zhipeng Li and Jian Chen and Peilin Zhao and Junzhou Huang},
  title = {{Towards Accurate and Compact Architectures via Neural Architecture Transformer}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3086914},
}
Figure