Introduction
In the field of Neural Architecture Search (NAS), research typically focuses on operation selection (e.g., convolutions vs. self-attention) or global depth. However, a critical yet often overlooked dimension is Layer Assignment: the distribution of a fixed number of layers across different resolution stages. For instance, a standard ResNet-50 distributes its 50 layers across four stages in a [3, 4, 6, 3] pattern. These distributions are traditionally handcrafted based on heuristics rather than empirical optimization. This paper introduces a novel approach to automate this design choice by identifying a fundamental property of network evolution.
The Neural Inheritance Relation (NIR)
The core contribution of this work is the discovery of the Neural Inheritance Relation (NIR). Through exhaustive experimentation on CIFAR-100, the authors observed that the optimal layer distribution for a deep network is almost always a direct "descendant" of the optimal distribution of a shallower version.
For example, if a 6-layer network performs best with a [2, 2, 2] assignment, the optimal 7-layer configuration is highly likely to be found among its immediate neighbors: [3, 2, 2], [2, 3, 2], or [2, 2, 3]. This observation suggests that successful architectural structures are inherited as depth increases, allowing for a structured exploration of the search space.
Methodology: Layer Assignment Search (LAS)
Leveraging the NIR, the authors propose Layer Assignment Search (LAS), a one-shot search method that utilizes Inherited Sampling. Instead of the standard approach of sampling configurations randomly from an exponential search space, LAS proceeds iteratively:
- Identify Shallow Optima: The search begins by finding the best-performing shallow assignments.
- Inherited Sampling: To evaluate the next depth level, the supernet only samples and trains the $n$ configurations that are direct descendants of the previous level's best assignment.
- One-Shot Evaluation: This process continues, significantly pruning the search space from $O(m^n)$ to $O(mn)$, where $m$ is depth and $n$ is the number of stages.
This method is remarkably efficient. Searching for an optimal MobileNet-V1 on ImageNet required only 86.8 GPU-hours—roughly half the time needed to train a single standalone model from scratch (173.9 hours).
Experimental Results
The authors validated LAS across several benchmarks, demonstrating that optimized layer assignments consistently outperform handcrafted defaults:
- ImageNet (MobileNet-V1): The searched assignment
[2, 3, 4, 3]achieved 71.32% Top-1 accuracy, a 1.5% improvement over the handcrafted[2, 2, 6, 2]baseline, while maintaining nearly identical FLOPs and parameter counts. - Tiny-ImageNet: LAS improved MobileNet-V1 accuracy by 1.42% and ResNet-50 by 0.4%.
- Consistency: The one-shot search results showed strong consistency with ground-truth optima found through exhaustive search on CIFAR-100, confirming the reliability of the NIR assumption.
Conclusion
This research demonstrates that "where" layers are placed is as vital as the total number of layers. By identifying the Neural Inheritance Relation, the authors provide a path to move beyond handcrafted heuristics toward a more principled, efficient method for backbone design. Future work may extend this inheritance logic to other architectural dimensions, such as channel width or kernel size, potentially leading to fully recursive evolutionary NAS frameworks.
Summary updated