TL;DR
This paper introduces NAT (Neural Architecture Transformer), a novel method for optimizing existing neural network architectures to improve performance and reduce computational cost. By framing architecture optimization as a Markov Decision Process and employing Graph Convolutional Networks, NAT effectively identifies and replaces redundant operations with more efficient ones, achieving state-of-the-art results on challenging image classification benchmarks.
What Problem Does This Paper Address?
Deep neural networks (DNNs) have achieved remarkable success across various domains, largely due to advancements in neural architecture design. However, designing these architectures is a labor-intensive process, often requiring significant human expertise. While Neural Architecture Search (NAS) automates this design process, the resulting architectures can still be suboptimal, containing redundant or computationally expensive operations. Existing optimization methods like NAO (Neural Architecture Optimization) can sometimes introduce more parameters or computation, and may produce architectures vastly different from the original. This paper addresses the need for an efficient and generalizable method to optimize existing architectures, making them more accurate and compact without sacrificing performance or introducing unnecessary complexity.
Methods
The core of this work is the Neural Architecture Transformer (NAT). The authors tackle the architecture optimization problem by casting it into a Markov Decision Process (MDP). In this framework, an architecture is considered a state, and transforming it into a more optimized version is an action. The goal is to learn a policy that maximizes the performance improvement while adhering to computational cost constraints.
To implement this, NAT uses a Graph Convolutional Network (GCN) to learn the optimization policy. The architecture is represented as a directed acyclic graph (DAG), where nodes represent feature maps and edges represent operations (like convolutions or skip connections). The GCN leverages the adjacency information of operations within the graph to predict optimal transformations. Specifically, NAT aims to replace expensive operations with more efficient ones, such as replacing a convolution with a skip connection or a null operation.
The training process involves an alternating scheme: first, the model parameters are trained with fixed NAT parameters, and then the NAT parameters are updated using policy gradient to maximize the expected performance improvement. An entropy regularization term is incorporated to encourage exploration and prevent premature convergence.
Datasets
The paper evaluates NAT on two widely-used image classification benchmark datasets:
- CIFAR-10: A dataset with 10 classes and 32x32 pixel images.
- ImageNet: A large-scale dataset with 1000 classes and high-resolution images.
Key Insights and Ideas
- Architecture Optimization as an MDP: Framing the problem as an MDP allows for a structured approach to decision-making about which operations to modify within an architecture. This is a novel way to think about improving existing models.
- GCNs for Policy Learning: Using GCNs to process the graph representation of an architecture is a key innovation. This allows NAT to effectively capture the local dependencies and context of operations, leading to better optimization decisions compared to sequential models like LSTMs.
- General-Purpose Optimizer: NAT is designed to be a general architecture optimizer, capable of taking any given architecture (hand-crafted or NAS-generated) as input and producing an optimized version. This broad applicability is a significant advantage.
- Balancing Accuracy and Compactness: The method aims to improve both performance (accuracy) and efficiency (fewer parameters, lower computation cost), addressing a critical trade-off in deep learning model development.
Key Results
NAT demonstrates significant improvements across various architectures:
- Hand-crafted Architectures:
- On ImageNet, NAT applied to VGG16 achieved a 2.75% improvement in Top-1 accuracy while maintaining a similar number of parameters and computation cost.
- For ResNet20 and MobileNetV2 on CIFAR-10, NAT also showed notable accuracy gains (e.g., 0.56% for ResNet20).
- NAS-based Architectures:
- On CIFAR-10, NAT-DARTS achieved 97.28% accuracy, outperforming the original DARTS (97.06%) and NAO-DARTS (97.09%) with 20% fewer parameters.
- On ImageNet, NAT-DARTS achieved 73.7% Top-1 accuracy, outperforming NAO-DARTS (73.3%) with fewer parameters and similar computation.
- NAT consistently improved upon baseline NAS models and those optimized by NAO, often with reduced parameter counts and computational overhead.
Limitations and Future Work
- Graph Representation Sensitivity: The authors acknowledge that the choice of graph representation for hand-crafted architectures can influence NAT's performance, and further research is needed to explore optimal representations.
- Computational Cost Evaluation: While NAT aims to reduce computation, the paper notes that accurately evaluating and bounding computation cost remains a challenging aspect of optimization problems.
- Future Work: The paper suggests that NAT can be extended to other tasks beyond image classification and could be further integrated with NAS methods for end-to-end architecture design and optimization. The generalizability of the GCN-based policy learner is also a promising avenue for future exploration.
Summary updated