Efficient Differentiable Neural Architecture Search with Meta Kernels

Shoufa Chen, Yunpeng Chen, Shuicheng Yan, Jiashi FengarXiv 2019 · arXiv

TL;DR

This paper introduces a breakthrough in Neural Architecture Search (NAS) by shifting the search process from weighting multiple feature maps to weighting convolutional kernels themselves. By merging candidate kernels into a single "Meta Kernel" before performing a convolution, the authors reduce search costs by three orders of magnitude while achieving state-of-the-art accuracy on ImageNet.

The Efficiency Bottleneck in NAS

Neural Architecture Search (NAS) aims to automate the design of deep learning models, but it is notoriously resource-heavy. Early methods used Reinforcement Learning or Evolutionary Algorithms, requiring thousands of GPU days.

Recent "differentiable" methods like DARTS made this faster by creating a supernet—a giant network where every layer contains multiple candidate operations (e.g., 3x3 conv, 5x5 conv, max pool). During the search, the supernet runs all these operations on the input and calculates a weighted sum of their outputs.

The bottleneck: Storing and calculating multiple feature maps for every single layer creates a massive memory and computation burden. This often forces researchers to search on tiny "proxy" datasets (like CIFAR-10) and hope the results transfer to larger ones (like ImageNet).

Methodology: Meta Kernels and Kernel Fusion

The authors propose a novel strategy called Meta Kernels based on the mathematical property of convolutional additivity.

Kernel-Level Fusion

Mathematically, if you have several convolution kernels of the same size operating on the same input, the sum of their outputs is equal to the output of the sum of the kernels. Instead of: $Output = (Input * Kernel_A) + (Input * Kernel_B)$ They do: $Output = Input * (Kernel_A + Kernel_B)$

Transition from weighting feature map outputs (left) to weighting convolutional kernels (right).

This allows the search process to perform only one convolution per layer instead of $N$. By summing weights/kernels first, the approach bypasses the memory explosion seen in traditional differentiable NAS.

Probabilistic Kernel Masks

To handle different kernel sizes (e.g., mixing a 3x3 and a 5x5), they use a "Meta Kernel" sized to the largest candidate. They then apply "Probabilistic Kernel Masks" that zero out the outer edges of the Meta Kernel to simulate smaller kernel sizes. This makes the discrete choice of kernel sizes differentiable via the Gumbel-Softmax trick, allowing the network to learn the best architecture using standard gradient descent.

Key Results: SOTA Accuracy at a Fraction of the Cost

The proposed method was evaluated primarily on ImageNet, the gold standard for large-scale image classification.

  • Massive Speedup: The search process is roughly 1,000x faster than MixNet and significantly faster than DARTS. Searching directly on ImageNet took only 40 GPU hours.
  • SOTA Performance: Their "metaKernel-B" model achieved 77.0% Top-1 accuracy on ImageNet with only 357M FLOPs. This outperforms both EfficientNet-B0 (76.3%) and MobileNetV3-Large (75.2%) at similar or lower computational costs.
Comparison of search cost and ImageNet accuracy across various NAS methods.

As shown in the figure above, the "ours" models (red stars) occupy the top-left region of the graph, representing the ideal combination of high accuracy and low search cost.

Insights and Transferability

Beyond raw metrics, the authors observed a fascinating trend: the search naturally chooses smaller kernels for shallow layers and larger kernels for deeper layers. This confirms the intuition that early layers should focus on local textures while deeper layers need a wider receptive field to understand complex objects.

Furthermore, when used as a backbone for object detection on PascalVOC (using YOLOv3), the Meta Kernel models outperformed established baselines like MobileNetV1 and V2 while maintaining a lower FLOP count, demonstrating strong transferability to downstream tasks.

Conclusion

By leveraging the mathematical properties of convolutions, Meta Kernels transform NAS from a memory-intensive operation into an efficient, direct search on target datasets. This approach effectively removes the need for proxy datasets and brings high-performance automated architecture design within reach of standard research hardware.

Summary updated

Abstract

The searching procedure of neural architecture search (NAS) is notoriously time consuming and cost prohibitive.To make the search space continuous, most existing gradient-based NAS methods relax the categorical choice of a particular operation to a softmax over all possible operations and calculate the weighted sum of multiple features, resulting in a large memory requirement and a huge computation burden. In this work, we propose an efficient and novel search strategy with meta kernels. We directly encode the supernet from the perspective on convolution kernels and "shrink" multiple convolution kernel candidates into a single one before these candidates operate on the input feature. In this way, only a single feature is generated between two intermediate nodes. The memory for storing intermediate features and the resource budget for conducting convolution operations are both reduced remarkably. Despite high efficiency, our search strategy can search in a more fine-grained way than existing works and increases the capacity for representing possible networks. We demonstrate the effectiveness of our search strategy by conducting extensive experiments. Specifically, our method achieves 77.0% top-1 accuracy on ImageNet benchmark dataset with merely 357M FLOPs, outperforming both EfficientNet and MobileNetV3 under the same FLOPs constraints. Compared to models discovered by the start-of-the-art NAS method, our method achieves the same (sometimes even better) performance, while faster by three orders of magnitude.
BibTeX
@misc{chen2019efficient,
  author = {Shoufa Chen and Yunpeng Chen and Shuicheng Yan and Jiashi Feng},
  title = {{Efficient Differentiable Neural Architecture Search with Meta Kernels}},
  year = {2019},
  eprint = {1912.04749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.04749},
}
Figure