TL;DR
Even state-of-the-art neural networks contain redundant operations that bloat computational cost and hinder performance. This paper introduces Neural Architecture Transformer (NAT) and its advanced version, NAT++, which automatically optimize any given architecture by replacing inefficient operations with lighter alternatives (like skip connections or separable convolutions) without increasing the overall compute budget. By framing this as a Markov Decision Process solved via a Graph Convolutional Network, the authors achieve higher accuracy and lower computational costs on both hand-crafted and NAS-generated models.
What Problem Does This Paper Address?
Designing neural network architectures is notoriously difficult. While human experts have crafted highly successful models (like ResNet and VGG) and Neural Architecture Search (NAS) algorithms have automated the discovery of others (like DARTS), the resulting architectures are rarely perfect. They often contain redundant or non-significant operations—such as unnecessary intermediate convolution or pooling layers.
This redundancy wastes memory and compute, and can even degrade predictive performance. Existing optimization methods, like Neural Architecture Optimization (NAO), attempt to fix this but often output entirely different architectures that are hard to analyze, or they improve accuracy at the expense of adding more parameters and computational overhead. The authors tackle the challenge of taking an existing architecture and strictly optimizing it to be more compact and accurate, without starting the search from scratch or increasing its computational cost.
Methods
The authors propose two iterative methods to transform architectures:
- Neural Architecture Transformer (NAT): NAT treats architecture optimization as a Markov Decision Process (MDP). It uses a Graph Convolutional Network (GCN) to "look" at the architecture's graph structure and decide which operations to replace with either a "skip connection" or a "null connection" (removing the edge entirely).
- NAT++: Recognizing that simply dropping operations limits the search space, the authors introduce NAT++. This version uses a two-level transition rule that allows for more granular downgrades. It can change the type of operation (e.g., standard convolution $\rightarrow$ separable convolution $\rightarrow$ pooling) and the kernel size (e.g., 5x5 $\rightarrow$ 3x3 $\rightarrow$ 1x1).
- Binary-Masked Softmax (BMSoftmax): To guarantee that the optimized architecture never exceeds the original's computational cost, NAT++ uses a custom BMSoftmax layer. This layer acts as a hard filter, zeroing out the probabilities of any "invalid" transitions that would increase the compute budget.
- Training: Because accuracy and computational cost are non-differentiable, the GCN controller is trained using Reinforcement Learning (Policy Gradient) to maximize the expected performance improvement on a validation set.
Datasets
The authors evaluate their methods across standard image classification and face recognition benchmarks:
- Image Classification: CIFAR-10, CIFAR-100, and ImageNet.
- Face Recognition: LFW, CFP-FP, and AgeDB-30.
Key Insights and Ideas
- Strict Cost Constraints Work: By strictly forcing the optimizer to only choose transitions that reduce or maintain computational cost (via the BMSoftmax layer), the model naturally discovers more efficient sub-graphs without needing complex, hard-to-tune penalty terms in the reward function.
- Granular Transitions Matter: NAT++ proves that simply dropping layers (as NAT does) leaves performance on the table. Allowing granular downgrades—like swapping a heavy 5x5 standard convolution for a lighter 3x3 separable convolution—provides a smoother optimization path and yields better architectures.
- Mitigating NAS Bias: A known flaw in NAS is that networks with many skip connections train faster in early epochs, tricking the search algorithm into preferring them despite poor final generalization. NAT/NAT++ avoids this trap by decoupling the supernet training from the search policy. By using uniform sampling during training, all operations get an equal chance to train, preventing the algorithm from unfairly favoring fast-converging skip connections.
Key Results
The proposed methods consistently improved both hand-crafted and automatically searched architectures:
- Hand-crafted models: On ImageNet, NAT++ boosted VGG16's Top-1 accuracy from 71.6% to 74.7% while reducing Multiply-Adds (MAdds) from 15.6G to 14.9G. ResNet50 improved from 76.2% to 77.8% with a slight reduction in compute.
- NAS models: When applied to the NAS-generated DARTS model on ImageNet, NAT++ boosted Top-1 accuracy from 73.1% to 73.9% while slashing computational cost by roughly 28% (574M MAdds down to 413M MAdds).
- Face Recognition: NAT++ successfully optimized LResNet34E-IR and MobileFaceNet, yielding state-of-the-art accuracy across LFW, CFP-FP, and AgeDB-30 without adding any computational overhead.
Limitations and Future Work
- Diminishing Returns on Highly Optimized Models: The authors note that NAT and NAT++ yield identical results when optimizing MobileNetV2. Because MobileNetV2 is already built almost entirely of highly efficient 1x1 convolutions and 3x3 separable convolutions, NAT++'s extended transition rules have no "cheaper" valid operations to transition to.
- GCN Depth Limits: The authors found that building a deeper GCN controller (e.g., 5 or 10 layers) actually degrades performance by introducing redundant information from multi-hop neighbors. They had to limit the GCN to 2 layers.
- Future Work: While not explicitly mapped out in a dedicated section, the paper implies that future work could involve automatically learning the transition rules (rather than relying on the hard-coded two-level rules) to better optimize already-compact architectures like MobileNetV2.
Summary updated