UNIT: Unifying Tensorized Instruction Compilation

Jian Weng, Animesh Jain, Jie Wang, Leyuan Wang, Yida Wang, Tony Nowatzki

TL;DR

UNIT (Unifying Tensorized Instruction Compilation) is a compiler framework that automates the use of specialized hardware instructions—like NVIDIA Tensor Cores or Intel VNNI—for deep learning. By abstracting these diverse hardware capabilities into a unified mathematical representation, UNIT allows developers to achieve performance that surpasses even highly optimized, hand-tuned vendor libraries like cuDNN and oneDNN.

What Problem Does This Paper Address?

To keep up with the massive computational demands of deep learning, hardware vendors have introduced "tensorized instructions" (e.g., Intel VNNI, NVIDIA Tensor Core, ARM DOT). These instructions are designed for mixed-precision math, where low-precision inputs (like int8) are multiplied and then accumulated into high-precision outputs (like int32) within a single hardware cycle.

However, these instructions are notoriously difficult to use. Standard compilers (like GCC or LLVM) aren't designed to handle the "horizontal accumulation" pattern these instructions require. Currently, developers have two suboptimal choices:

  1. Vendor Libraries (cuDNN, oneDNN): These are fast but act as "black boxes." They only work for specific, pre-defined operations and are difficult to optimize further or adapt to new model architectures.
  2. Manual Intrinsics: Writing assembly-level code for every different chip. This is error-prone, time-consuming, and must be redone for every new hardware generation.

UNIT addresses this "compilation gap" by providing a way to automatically detect, map, and optimize these instructions across different hardware platforms.

Methods

The authors built UNIT on top of the Apache TVM compiler stack. The framework consists of three main stages:

  • Unified Semantics Abstraction: Instead of writing custom logic for every chip, UNIT uses a Tensor Domain-Specific Language (DSL) to describe what a hardware instruction does. If an instruction performs a dot product, it’s described as a small loop nest in the DSL.
  • The Inspector (Applicability Detection): This component analyzes a high-level deep learning operation (like a convolution) and the hardware instruction's DSL. It checks for arithmetic isomorphism—essentially proving that the math in the model matches the math the hardware can do. It also verifies that data can be moved into the registers in a way the hardware expects.
  • The Rewriter & Tuner: Once a match is found, the Rewriter tiles and reorders the loops of the original operation so the innermost parts perfectly fit the hardware instruction. The Tuner then explores thousands of ways to organize the outer loops (handling threading, cache reuse, and unrolling) to find the fastest possible execution strategy for that specific chip.

Datasets

The researchers evaluated UNIT using a variety of standard Deep Learning benchmarks and workloads:

  • Models: ResNet-18, ResNet-50, Inception-v3, MobileNet-v1, and MobileNet-v2.
  • Workloads: 148 different convolution layers representing diverse shapes, strides, and channel depths.
  • Hardware Platforms:
    • Intel x86 CPU: AWS EC2 instance with Cascade Lake (VNNI support).
  • NVIDIA GPU: Tesla V100 (Tensor Core support).
  • ARM CPU: AWS Graviton2 (ARM DOT support).

Key Insights and Ideas

  • Isomorphism over Integration: The core "aha moment" is that you don't need to write a new compiler for every new chip. By treating hardware instructions as tiny programs (DSL), the problem of using them becomes a search for mathematical equivalence (isomorphism).
  • Decoupling Logic from Optimization: UNIT separates the correctness of using an instruction from the performance of the surrounding code. This allows the framework to be general enough for any chip while still being fast enough to beat hand-tuned code.
  • Handling "Horizontal" Math: Unlike standard SIMD (Single Instruction, Multiple Data) which performs the same operation on independent lanes, tensorized instructions reduce multiple lanes into one. UNIT’s ability to automatically reorganize loop nests to satisfy this reduction dependency is a significant technical leap.

Key Results

UNIT consistently outperformed industry-standard vendor libraries across all tested platforms:

  • Intel x86: Achieved 1.3× speedup over Intel’s own oneDNN.
  • NVIDIA GPU: Achieved 1.75× speedup (and up to 2.2×) over cuDNN.
  • ARM CPU: Achieved 1.13× speedup over a manually tuned TVM implementation for ARM DOT.
  • Extensibility: To prove the system's flexibility, the authors applied UNIT to 3D Convolutions. Without changing the compiler's core logic, UNIT successfully mapped the 3D math to 1D hardware instructions, outperforming oneDNN by 1.2×.

Limitations and Future Work

  • Manual Operand Mapping: While the loop transformation is automatic, a developer still needs to manually define the "rule of operand generation" (how data is packed into registers) when adding a brand-new hardware platform to the system.
  • Adversarial Strides: The authors noted that UNIT performs slightly worse on specific layers with unusual "strided" data access patterns, which can hurt data locality.
  • Scope: The current work focuses on dense tensor operations (Convolutions and Matrix Multiplications). Future work could extend this unified approach to sparse operations or more exotic hardware accelerators beyond general-purpose CPUs and GPUs.

Summary updated

Abstract

Because of the increasing demand for intensive computation in deep neural networks, researchers have developed both hardware and software mechanisms to reduce the compute and memory burden. A widely adopted approach is to use mixed precision data types. However, it is hard to benefit from mixed precision without hardware specialization because of the overhead of data casting. Recently, hardware vendors offer tensorized instructions specialized for mixed-precision tensor operations, such as Intel VNNI, Nvidia Tensor Core, and ARM DOT. These instructions involve a new computing idiom, which reduces multiple low precision elements into one high precision element. The lack of compilation techniques for this emerging idiom makes it hard to utilize these instructions. In practice, one approach is to use vendor-provided libraries for computationally-intensive kernels, but this is inflexible and prevents further optimizations. Another approach is to manually write hardware intrinsics, which is error-prone and difficult for programmers. Some prior works tried to address this problem by creating compilers for each instruction. This requires excessive efforts when it comes to many tensorized instructions. In this work, we develop a compiler framework, UNIT, to unify the compilation for tensorized instructions. The key to this approach is a unified semantics abstraction which makes the integration of new instructions easy, and the reuse of the analysis and transformations possible. Tensorized instructions from different platforms can be compiled via UNIT with moderate effort for favorable performance. Given a tensorized instruction and a tensor operation, UNIT automatically detects the applicability of the instruction, transforms the loop organization of the operation, and rewrites the loop body to take advantage of the tensorized instruction. According to our evaluation, UNIT is able to target various mainstream hardware platforms. The generated end-to-end inference model achieves 1.3 x speedup over Intel oneDNN on an x86 CPU, 1.75x speedup over Nvidia cuDNN on an Nvidia GPU, and 1.13x speedup over a carefully tuned TVM solution for ARM DOT on an ARM CPU.
BibTeX
@inproceedings{weng2021unit,
  author = {Jian Weng and Animesh Jain and Jie Wang and Leyuan Wang and Yida Wang and Tony Nowatzki},
  title = {{UNIT: Unifying Tensorized Instruction Compilation}},
  booktitle = {Cgo},
  year = {2021},
  doi = {10.1109/cgo51591.2021.9370330},
}

Loading related content...

Figure