Neural Inheritance Relation Guided One-Shot Layer Assignment Search

Rang Meng, Weijie Chen, Di Xie, Yuan Zhang, Shiliang Pu

Introduction

In the field of Neural Architecture Search (NAS), research typically focuses on operation selection (e.g., convolutions vs. self-attention) or global depth. However, a critical yet often overlooked dimension is Layer Assignment: the distribution of a fixed number of layers across different resolution stages. For instance, a standard ResNet-50 distributes its 50 layers across four stages in a [3, 4, 6, 3] pattern. These distributions are traditionally handcrafted based on heuristics rather than empirical optimization. This paper introduces a novel approach to automate this design choice by identifying a fundamental property of network evolution.

The Neural Inheritance Relation (NIR)

The core contribution of this work is the discovery of the Neural Inheritance Relation (NIR). Through exhaustive experimentation on CIFAR-100, the authors observed that the optimal layer distribution for a deep network is almost always a direct "descendant" of the optimal distribution of a shallower version.

The Neural Inheritance Relation tree visualizes how optimal assignments (yellow) at one depth level serve as the foundation for the best configurations at the next depth level.

For example, if a 6-layer network performs best with a [2, 2, 2] assignment, the optimal 7-layer configuration is highly likely to be found among its immediate neighbors: [3, 2, 2], [2, 3, 2], or [2, 2, 3]. This observation suggests that successful architectural structures are inherited as depth increases, allowing for a structured exploration of the search space.

Methodology: Layer Assignment Search (LAS)

Leveraging the NIR, the authors propose Layer Assignment Search (LAS), a one-shot search method that utilizes Inherited Sampling. Instead of the standard approach of sampling configurations randomly from an exponential search space, LAS proceeds iteratively:

  1. Identify Shallow Optima: The search begins by finding the best-performing shallow assignments.
  2. Inherited Sampling: To evaluate the next depth level, the supernet only samples and trains the $n$ configurations that are direct descendants of the previous level's best assignment.
  3. One-Shot Evaluation: This process continues, significantly pruning the search space from $O(m^n)$ to $O(mn)$, where $m$ is depth and $n$ is the number of stages.
The Inherited Sampling mechanism in a one-shot supernet uses prior optimal layers to focus training and evaluation on the most promising descendant architectures.

This method is remarkably efficient. Searching for an optimal MobileNet-V1 on ImageNet required only 86.8 GPU-hours—roughly half the time needed to train a single standalone model from scratch (173.9 hours).

Experimental Results

The authors validated LAS across several benchmarks, demonstrating that optimized layer assignments consistently outperform handcrafted defaults:

  • ImageNet (MobileNet-V1): The searched assignment [2, 3, 4, 3] achieved 71.32% Top-1 accuracy, a 1.5% improvement over the handcrafted [2, 2, 6, 2] baseline, while maintaining nearly identical FLOPs and parameter counts.
  • Tiny-ImageNet: LAS improved MobileNet-V1 accuracy by 1.42% and ResNet-50 by 0.4%.
  • Consistency: The one-shot search results showed strong consistency with ground-truth optima found through exhaustive search on CIFAR-100, confirming the reliability of the NIR assumption.

Conclusion

This research demonstrates that "where" layers are placed is as vital as the total number of layers. By identifying the Neural Inheritance Relation, the authors provide a path to move beyond handcrafted heuristics toward a more principled, efficient method for backbone design. Future work may extend this inheritance logic to other architectural dimensions, such as channel width or kernel size, potentially leading to fully recursive evolutionary NAS frameworks.

Summary updated

Abstract

Layer assignment is seldom picked out as an independent research topic in neural architecture search. In this paper, for the first time, we systematically investigate the impact of different layer assignments to the network performance by building an architecture dataset of layer assignment on CIFAR-100. Through analyzing this dataset, we discover a neural inheritance relation among the networks with different layer assignments, that is, the optimal layer assignments for deeper networks always inherit from those for shallow networks. Inspired by this neural inheritance relation, we propose an efficient one-shot layer assignment search approach via inherited sampling. Specifically, the optimal layer assignment searched in the shallow network can be provided as a strong sampling priori to train and search the deeper ones in supernet, which extremely reduces the network search space. Comprehensive experiments carried out on CIFAR-100 illustrate the efficiency of our proposed method. Our search results are strongly consistent with the optimal ones directly selected from the architecture dataset. To further confirm the generalization of our proposed method, we also conduct experiments on Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the handcrafted ones under the unchanged computational budgets. The neural inheritance relation discovered in this paper can provide insights to the universal neural architecture search.
BibTeX
@inproceedings{meng2020neural,
  author = {Rang Meng and Weijie Chen and Di Xie and Yuan Zhang and Shiliang Pu},
  title = {{Neural Inheritance Relation Guided One-Shot Layer Assignment Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5959},
}
Figure