Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance

Jiarong Xing, Leyuan Wang, Shang Zhang, Jack Chen, Ang Chen, Yibo Zhu

TL;DR

Deep learning auto-tuners like Ansor often struggle to match hand-tuned vendor libraries on modern GPU hardware (such as Tensor Cores) and can take hours or days to search for optimal configurations. Bolt bridges this gap by marrying auto-tuning with modern, modularized template libraries (specifically NVIDIA CUTLASS) within TVM. By employing a lightweight hardware-aware profiler and novel "persistent kernel fusion," Bolt achieves up to 2.5× faster inference on common CNNs compared to Ansor while slashing tuning time from ~12 hours down to under 20 minutes.

What Problem Does This Paper Address?

Modern deep learning compilers rely heavily on auto-tuners (e.g., AutoTVM, Ansor) to generate fast execution schedules. To stay hardware-agnostic, these auto-tuners treat devices as "black boxes" and learn cost models from scratch. However, this opaque approach causes two major problems:

  1. Suboptimal performance on specialized hardware: They struggle to effectively utilize specialized execution units like NVIDIA Tensor Cores. For instance, on FP16 matrix multiplications (GEMMs), Ansor achieves less than 20% of cuBLAS performance on an NVIDIA Tesla T4 GPU.
  2. Exhaustive tuning time: Navigating massive, hardware-unaware search spaces requires searching thousands of configurations, taking days for complex models and failing to adapt quickly to dynamic batch sizes or shapes.

While vendor libraries (cuBLAS, cuDNN) provide native speed, they offer rigid, fixed APIs that do not support arbitrary operator fusion or rapid compiler customization.

Methods

Bolt leverages the emerging trend of templated vendor libraries (e.g., NVIDIA CUTLASS), which are modular and parameterizable in C++ templates:

  • Automated Templated Code Generation & Profiler: Bolt integrates into TVM using a Bring Your Own Codegen (BYOC) workflow. Instead of searching a massive space, Bolt's profiler encodes hardware guidelines (e.g., register limits, optimal warp tile sizes, alignment) to narrow the search to tens of high-quality parameter combinations, evaluating them directly with concrete inputs in minutes.
  • Persistent Kernel Fusion (Deeper Operator Fusion): Existing compilers typically only fuse a GEMM/Conv with subsequent element-wise layers (epilogue fusion). Bolt extends CUTLASS templates to fuse consecutive GEMMs or Convs (e.g., $1 \times 1$ Conv following a $3 \times 3$ Conv) into a single GPU kernel. It enforces threadblock residence—keeping intermediate activations strictly on-chip—using either:
  • Register File (RF)-resident fusion: Intermediate data stays in warp accumulator registers.
  • Shared Memory (SMEM)-resident fusion: Intermediate data stages through shared memory with bank-conflict-free layouts when output channel dimensions require inter-warp communication.
  • Layout and Alignment Optimizations: Bolt automatically enforces NHWC layouts (faster in CUTLASS) and pads non-8-divisible tensor channels to alignment 8 to maximize 128-bit vectorized memory accesses.

Datasets

  • ImageNet: Used for training and evaluating accuracy in the RepVGG case study models (RepVGG-A0, A1, B0) trained with FP32 and quantized to FP16.
  • Operator & Model Benchmarks: Workloads extracted from BERT (NLP), ResNet-50, VGG-16/19, and recommendation models (DCNv2, DLRM).

Key Insights and Ideas

  • Templates as an Abstraction Layer: Parameterized C++ template libraries provide an ideal middle ground between black-box auto-scheduling and rigid vendor binaries, exposing low-level hardware optimizations while remaining composable.
  • Register/SRAM-Level Multi-Operator Fusion: By keeping intermediate activations in GPU registers or shared memory across back-to-back GEMM/Conv layers, Bolt completely eliminates intermediate global memory round-trips and reduces kernel launch overheads.
  • System-Model Codesign: Understanding hardware template constraints reveals clear model design guidelines:
  • Fusing activation functions via epilogues makes switching activations (e.g., ReLU to Hardswish) virtually free in latency.
  • Adding $1 \times 1$ convolutions increases network depth and accuracy with minimal latency penalty due to persistent kernel fusion.
  • Aligning channel dimensions to multiples of 8 avoids costly runtime padding overheads (which otherwise took ~16% of total compute time in benchmarks).

Key Results

  • End-to-End Inference Speed: Bolt delivers an average 2.5× speedup (up to 2.8× across tested vision networks, including 4.2× on VGG models, 1.5× on ResNet, and 2.6× on RepVGG) over Ansor on an NVIDIA Tesla T4 GPU.
  • Tuning Turnaround: Slashed auto-tuning time from ~12 hours (Ansor) to under 20 minutes.
  • Microbenchmarks:
  • FP16 GEMMs achieved 6.1–9.5× speedup over Ansor on compute-heavy workloads.
  • Persistent kernel fusion accelerated back-to-back GEMMs by 1.2–1.5× and back-to-back Convs by 1.1–2.0× over sequential execution.
  • Channel padding to 8-element alignment improved Conv2D speed by 1.8×.
  • RepVGG Case Study: Augmenting RepVGG-A1 with $1 \times 1$ Convs and Hardswish improved ImageNet Top-1 accuracy by 1.83% with comparable latency to baseline designs.

Limitations and Future Work

  • Operator Scope: Persistent kernel fusion is strictly beneficial for memory-bound operations (e.g., small channel or matrix dimensions $N$ and $K$ with large batch $M$). For compute-bound operators, enforcing threadblock residence restricts tiling options and can degrade performance.
  • Platform Scope: The evaluation is centered on NVIDIA GPUs and CUTLASS. While the concepts transfer to other templated systems (such as Intel OneDNN and AMD ROCm), implementations on those platforms were left for future work.

Summary updated

Abstract

Today's auto-tuners (e.g., AutoTVM, Ansor) generate efficient tensor programs by navigating a large search space to identify effective implementations, but they do so with opaque hardware details. Thus, their performance could fall behind that of hardware-native libraries (e.g., cuBLAS, cuDNN), which are hand-optimized by device vendors to extract high performance. On the other hand, these vendor libraries have a fixed set of supported functions and lack the customization and automation support afforded by auto-tuners. Bolt is based on the recent trend that vendor libraries are increasingly modularized and reconfigurable via declarative control (e.g., CUTLASS). It enables a novel approach that bridges this gap and achieves the best of both worlds, via hardware-native templated search. Bolt provides new opportunities to rethink end-to-end tensor optimizations at the graph, operator, and model levels. Bolt demonstrates this concept by prototyping on a popular auto-tuner in TVM and a class of widely-used platforms (i.e., NVIDIA GPUs) -- both in large deployment in our production environment. Bolt improves the inference speed of common convolutional neural networks by 2.5x on average over the state of the art, and it auto-tunes these models within 20 minutes.
BibTeX
@inproceedings{xing2022bolt,
  author = {Jiarong Xing and Leyuan Wang and Shang Zhang and Jack Chen and Ang Chen and Yibo Zhu},
  title = {{Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance}},
  booktitle = {MLSys},
  year = {2022},
  url = {https://openreview.net/forum?id=qmIETUd83c},
}

Loading related content...

Figure