TL;DR
This paper introduces a breakthrough in Neural Architecture Search (NAS) by shifting the search process from weighting multiple feature maps to weighting convolutional kernels themselves. By merging candidate kernels into a single "Meta Kernel" before performing a convolution, the authors reduce search costs by three orders of magnitude while achieving state-of-the-art accuracy on ImageNet.
The Efficiency Bottleneck in NAS
Neural Architecture Search (NAS) aims to automate the design of deep learning models, but it is notoriously resource-heavy. Early methods used Reinforcement Learning or Evolutionary Algorithms, requiring thousands of GPU days.
Recent "differentiable" methods like DARTS made this faster by creating a supernet—a giant network where every layer contains multiple candidate operations (e.g., 3x3 conv, 5x5 conv, max pool). During the search, the supernet runs all these operations on the input and calculates a weighted sum of their outputs.
The bottleneck: Storing and calculating multiple feature maps for every single layer creates a massive memory and computation burden. This often forces researchers to search on tiny "proxy" datasets (like CIFAR-10) and hope the results transfer to larger ones (like ImageNet).
Methodology: Meta Kernels and Kernel Fusion
The authors propose a novel strategy called Meta Kernels based on the mathematical property of convolutional additivity.
Kernel-Level Fusion
Mathematically, if you have several convolution kernels of the same size operating on the same input, the sum of their outputs is equal to the output of the sum of the kernels. Instead of: $Output = (Input * Kernel_A) + (Input * Kernel_B)$ They do: $Output = Input * (Kernel_A + Kernel_B)$
This allows the search process to perform only one convolution per layer instead of $N$. By summing weights/kernels first, the approach bypasses the memory explosion seen in traditional differentiable NAS.
Probabilistic Kernel Masks
To handle different kernel sizes (e.g., mixing a 3x3 and a 5x5), they use a "Meta Kernel" sized to the largest candidate. They then apply "Probabilistic Kernel Masks" that zero out the outer edges of the Meta Kernel to simulate smaller kernel sizes. This makes the discrete choice of kernel sizes differentiable via the Gumbel-Softmax trick, allowing the network to learn the best architecture using standard gradient descent.
Key Results: SOTA Accuracy at a Fraction of the Cost
The proposed method was evaluated primarily on ImageNet, the gold standard for large-scale image classification.
- Massive Speedup: The search process is roughly 1,000x faster than MixNet and significantly faster than DARTS. Searching directly on ImageNet took only 40 GPU hours.
- SOTA Performance: Their "metaKernel-B" model achieved 77.0% Top-1 accuracy on ImageNet with only 357M FLOPs. This outperforms both EfficientNet-B0 (76.3%) and MobileNetV3-Large (75.2%) at similar or lower computational costs.
As shown in the figure above, the "ours" models (red stars) occupy the top-left region of the graph, representing the ideal combination of high accuracy and low search cost.
Insights and Transferability
Beyond raw metrics, the authors observed a fascinating trend: the search naturally chooses smaller kernels for shallow layers and larger kernels for deeper layers. This confirms the intuition that early layers should focus on local textures while deeper layers need a wider receptive field to understand complex objects.
Furthermore, when used as a backbone for object detection on PascalVOC (using YOLOv3), the Meta Kernel models outperformed established baselines like MobileNetV1 and V2 while maintaining a lower FLOP count, demonstrating strong transferability to downstream tasks.
Conclusion
By leveraging the mathematical properties of convolutions, Meta Kernels transform NAS from a memory-intensive operation into an efficient, direct search on target datasets. This approach effectively removes the need for proxy datasets and brings high-performance automated architecture design within reach of standard research hardware.
Summary updated