TL;DR
This paper introduces a novel Neural Architecture Search (NAS) method that addresses the fairness and interpretability issues in weight-sharing NAS. By leveraging the concept of "dynamical isometry" and a new initialization strategy, the authors ensure that candidate network modules are evaluated fairly, leading to improved search efficiency and state-of-the-art accuracy on ImageNet.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) aims to automate the design of neural networks. Weight-sharing NAS methods significantly speed up this process by training a single "supernet" that contains all candidate architectures. However, these methods suffer from two main problems:
- Lack of Fairness and Interpretability: The weights of candidate subnets are highly coupled, making it difficult to evaluate their individual performance objectively. Existing methods often rely on empirical indicators (like Batch Normalization layer weights) without strong theoretical guarantees, leading to potentially irrational search results.
- Module Evaluation Dilemma: Even with indicators like BN weights, comparing modules across different layers is problematic because their input signals can vary significantly, especially in deeper layers. This makes it hard to rank modules fairly.
This paper tackles these issues by proposing a method that ensures "rigorous fairness" in the search process, making it more interpretable and efficient.
Methods
The core of the proposed method lies in two key innovations:
- Dynamical Isometry Initialization: The authors propose a new initialization strategy for the supernet's weights. This strategy aims to achieve "dynamical isometry," a property where the input-output Jacobian matrix of each module has singular values close to 1. This ensures that signals propagate stably through the network without vanishing or exploding, and crucially, that each candidate module receives equivalent input signals, regardless of its depth. This is achieved by triangularly decomposing the randomly initialized weights.
- Rigorous Fair Module Evaluation: Building on dynamical isometry, the paper theoretically proves that the learnable parameters of Batch Normalization (BN) layers can serve as a rational and interpretable criterion for evaluating candidate modules. By ensuring dynamical isometry, the BN layer's parameters (specifically, their learned weights) are shown to reflect the signal propagation capability of the module, thus providing a fair basis for comparison. The paper also introduces a strategy to address the issue of comparing BN indicators across different layers.
Datasets
- ImageNet: Used for training and evaluating the searched neural network architectures.
Key Insights and Ideas
- Dynamical Isometry as a Foundation for Fairness: The paper establishes a strong theoretical link between dynamical isometry and fair evaluation in NAS. By ensuring stable signal propagation, it creates an environment where modules can be compared on a level playing field.
- BN Layer Parameters as Interpretable Indicators: The work provides a mathematical proof that BN layer parameters can be a theoretically sound indicator of module performance, moving beyond empirical heuristics.
- Addressing Layer-wise Comparison: The proposed method tackles the challenge of comparing module performance across different layers, a significant hurdle in existing weight-sharing NAS.
- Efficiency Gains: The orthogonal initialization strategy, combined with the focus on BN layers during training, leads to substantial speedups in the supernet training phase.
Key Results
- State-of-the-Art Accuracy: Architectures searched using the proposed method achieve state-of-the-art top-1 validation accuracy on ImageNet, comparable to or exceeding existing methods, while using the same model size.
- Improved Training Stability: The method demonstrates better and more stable training performance.
- Significant Speedup: The supernet training time is dramatically reduced compared to other methods, with each image taking less propagation time. For example, the supernet training time is reported as 2.8 GPU days for 80 epochs.
- Competitive Latency: The proposed method's latency (93.1 ms) narrowly outperforms BNNAS (96.3 ms) in comparative experiments.
Limitations and Future Work
- Untrained Network Parameters: The paper acknowledges that since network parameters are untrained, it's almost impossible to achieve the expected state of a candidate module perfectly. The method aims to minimize deviations by adjusting parameters.
- Future Work: The paper suggests that further research could explore the generalization ability of the method by changing search spaces and investigating the impact of adding BN-based layer indicators. The authors also imply that their framework can be used to find multiple excellent models from a search space simultaneously.
Summary updated