TL;DR
UNIT (Unifying Tensorized Instruction Compilation) is a compiler framework that automates the use of specialized hardware instructions—like NVIDIA Tensor Cores or Intel VNNI—for deep learning. By abstracting these diverse hardware capabilities into a unified mathematical representation, UNIT allows developers to achieve performance that surpasses even highly optimized, hand-tuned vendor libraries like cuDNN and oneDNN.
What Problem Does This Paper Address?
To keep up with the massive computational demands of deep learning, hardware vendors have introduced "tensorized instructions" (e.g., Intel VNNI, NVIDIA Tensor Core, ARM DOT). These instructions are designed for mixed-precision math, where low-precision inputs (like int8) are multiplied and then accumulated into high-precision outputs (like int32) within a single hardware cycle.
However, these instructions are notoriously difficult to use. Standard compilers (like GCC or LLVM) aren't designed to handle the "horizontal accumulation" pattern these instructions require. Currently, developers have two suboptimal choices:
- Vendor Libraries (cuDNN, oneDNN): These are fast but act as "black boxes." They only work for specific, pre-defined operations and are difficult to optimize further or adapt to new model architectures.
- Manual Intrinsics: Writing assembly-level code for every different chip. This is error-prone, time-consuming, and must be redone for every new hardware generation.
UNIT addresses this "compilation gap" by providing a way to automatically detect, map, and optimize these instructions across different hardware platforms.
Methods
The authors built UNIT on top of the Apache TVM compiler stack. The framework consists of three main stages:
- Unified Semantics Abstraction: Instead of writing custom logic for every chip, UNIT uses a Tensor Domain-Specific Language (DSL) to describe what a hardware instruction does. If an instruction performs a dot product, it’s described as a small loop nest in the DSL.
- The Inspector (Applicability Detection): This component analyzes a high-level deep learning operation (like a convolution) and the hardware instruction's DSL. It checks for arithmetic isomorphism—essentially proving that the math in the model matches the math the hardware can do. It also verifies that data can be moved into the registers in a way the hardware expects.
- The Rewriter & Tuner: Once a match is found, the Rewriter tiles and reorders the loops of the original operation so the innermost parts perfectly fit the hardware instruction. The Tuner then explores thousands of ways to organize the outer loops (handling threading, cache reuse, and unrolling) to find the fastest possible execution strategy for that specific chip.
Datasets
The researchers evaluated UNIT using a variety of standard Deep Learning benchmarks and workloads:
- Models: ResNet-18, ResNet-50, Inception-v3, MobileNet-v1, and MobileNet-v2.
- Workloads: 148 different convolution layers representing diverse shapes, strides, and channel depths.
- Hardware Platforms:
- Intel x86 CPU: AWS EC2 instance with Cascade Lake (VNNI support).
- NVIDIA GPU: Tesla V100 (Tensor Core support).
- ARM CPU: AWS Graviton2 (ARM DOT support).
Key Insights and Ideas
- Isomorphism over Integration: The core "aha moment" is that you don't need to write a new compiler for every new chip. By treating hardware instructions as tiny programs (DSL), the problem of using them becomes a search for mathematical equivalence (isomorphism).
- Decoupling Logic from Optimization: UNIT separates the correctness of using an instruction from the performance of the surrounding code. This allows the framework to be general enough for any chip while still being fast enough to beat hand-tuned code.
- Handling "Horizontal" Math: Unlike standard SIMD (Single Instruction, Multiple Data) which performs the same operation on independent lanes, tensorized instructions reduce multiple lanes into one. UNIT’s ability to automatically reorganize loop nests to satisfy this reduction dependency is a significant technical leap.
Key Results
UNIT consistently outperformed industry-standard vendor libraries across all tested platforms:
- Intel x86: Achieved 1.3× speedup over Intel’s own oneDNN.
- NVIDIA GPU: Achieved 1.75× speedup (and up to 2.2×) over cuDNN.
- ARM CPU: Achieved 1.13× speedup over a manually tuned TVM implementation for ARM DOT.
- Extensibility: To prove the system's flexibility, the authors applied UNIT to 3D Convolutions. Without changing the compiler's core logic, UNIT successfully mapped the 3D math to 1D hardware instructions, outperforming oneDNN by 1.2×.
Limitations and Future Work
- Manual Operand Mapping: While the loop transformation is automatic, a developer still needs to manually define the "rule of operand generation" (how data is packed into registers) when adding a brand-new hardware platform to the system.
- Adversarial Strides: The authors noted that UNIT performs slightly worse on specific layers with unusual "strided" data access patterns, which can hurt data locality.
- Scope: The current work focuses on dense tensor operations (Convolutions and Matrix Multiplications). Future work could extend this unified approach to sparse operations or more exotic hardware accelerators beyond general-purpose CPUs and GPUs.
Summary updated