Canvas: End-to-End Kernel Architecture Search in Neural Networks

Chenggang Zhao, Genghan Zhang, Mingyu Gao

Abstract

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations.

Introduction

In modern deep learning development, a significant divide exists between machine learning researchers and systems engineers. ML researchers typically focus on Neural Architecture Search (NAS) to optimize high-level connectivity between fixed blocks (e.g., convolutions) for accuracy. Conversely, systems engineers utilize tensor compilers like TVM to optimize the execution of these fixed blocks on hardware. Both approaches treat the "kernel"—the internal mathematical operation—as a static black box.

Canvas introduces Kernel Architecture Search (KAS) to bridge this gap. By relaxing the requirement for strict mathematical equivalence, KAS allows for the redesign of the fundamental "atoms" of a neural network. This approach enables the discovery of new kernel structures that optimize for both hardware performance and model accuracy simultaneously.

The Canvas Framework

Canvas is an end-to-end framework designed to automate the discovery of high-performance kernels. The system operates through a multi-stage pipeline that transforms an "empty kernel" into a fully optimized, hardware-ready implementation.

The Canvas end-to-end search and evaluation pipeline.

The process begins with a Micro-DAG Random Sampler that stochastically explores combinations of fine-grained primitives. Because merging different computational branches often leads to dimension mismatches, Canvas employs a two-level solver system:

  1. Solver #1 (Shape Matching): Ensures structural legality by resolving tensor dimension constraints.
  2. Solver #2 (Constraint Solver): Tunes hyperparameters, such as channel counts, to ensure the resulting kernel fits within specified FLOP or parameter budgets.

Finally, the generated kernels are passed through a code optimization and generation phase (using TVM) and evaluated against latency and accuracy targets.

Search Space and Primitives

The core of Canvas's flexibility lies in its library of fine-grained primitives. Instead of relying on monolithic convolution operations, Canvas decomposes kernels into simpler building blocks categorized into three main classes: Rearrangement, Arithmetic, and Blending.

Fine-grained primitives used as building blocks for kernel generation.

By decoupling data movement (rearrangement) from computation (arithmetic), Canvas can synthesize complex behaviors. For example, it can mimic a 3x3 convolution by first shifting pixels and then applying a 1x1 convolution, achieving similar receptive fields at a significantly lower computational cost.

Evaluation and Results

The authors evaluated Canvas across eight popular backbones, including ResNet, VGG, and MobileNet-V2, using CIFAR-100 for proxy search and ImageNet for final validation.

  • Performance Gains: Canvas-generated models achieved an average 1.5x speedup over state-of-the-art TVM Ansor optimizations, with peak improvements reaching 4.6x on VGG-16.
  • Efficiency: The search process naturally favored lightweight designs, resulting in model size reductions of 40% to 90%.
  • Accuracy: These performance leaps were achieved with manageable accuracy trade-offs, typically around 1% on CIFAR-100 and 4% on ImageNet—comparable to aggressive pruning techniques.

Discussion and Conclusion

A compelling finding of the KAS approach is its ability to "rediscover" human-designed innovations like Depth-wise Separable Convolutions and Squeeze-and-Excitation blocks. Furthermore, Canvas identifies "irregular" structures—asymmetric branches that humans might overlook—which prove highly efficient on modern hardware.

While Canvas currently requires retraining due to the lack of mathematical equivalence and focuses primarily on convolutional replacements, it demonstrates the immense potential of KAS. Future work aims to extend this paradigm to Attention mechanisms and specialized AI accelerators, further blurring the line between architecture design and system optimization.

Summary updated

Abstract

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations. For source code and implementation, we open-sourced Canvas at https://github.com/tsinghua-ideal/Canvas.
BibTeX
@misc{zhao2023canvas,
  author = {Chenggang Zhao and Genghan Zhang and Mingyu Gao},
  title = {{Canvas: End-to-End Kernel Architecture Search in Neural Networks}},
  year = {2023},
  eprint = {2304.07741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.07741},
}

Loading related content...

Figure