Abstract
Because of the increasing demand for intensive computation in deep neural networks, researchers have developed both hardware and software mechanisms to reduce the compute and memory burden. A widely adopted approach is to use mixed precision data types. However, it is hard to benefit from mixed precision without hardware specialization because of the overhead of data casting. Recently, hardware vendors offer tensorized instructions specialized for mixed-precision tensor operations, such as Intel VNNI, Nvidia Tensor Core, and ARM DOT. These instructions involve a new computing idiom, which reduces multiple low precision elements into one high precision element. The lack of compilation techniques for this emerging idiom makes it hard to utilize these instructions. In practice, one approach is to use vendor-provided libraries for computationally-intensive kernels, but this is inflexible and prevents further optimizations. Another approach is to manually write hardware intrinsics, which is error-prone and difficult for programmers. Some prior works tried to address this problem by creating compilers for each instruction. This requires excessive efforts when it comes to many tensorized instructions. In this work, we develop a compiler framework, UNIT, to unify the compilation for tensorized instructions. The key to this approach is a unified semantics abstraction which makes the integration of new instructions easy, and the reuse of the analysis and transformations possible. Tensorized instructions from different platforms can be compiled via UNIT with moderate effort for favorable performance. Given a tensorized instruction and a tensor operation, UNIT automatically detects the applicability of the instruction, transforms the loop organization of the operation, and rewrites the loop body to take advantage of the tensorized instruction. According to our evaluation, UNIT is able to target various mainstream hardware platforms. The generated end-to-end inference model achieves 1.3 x speedup over Intel oneDNN on an x86 CPU, 1.75x speedup over Nvidia cuDNN on an Nvidia GPU, and 1.13x speedup over a carefully tuned TVM solution for ARM DOT on an ARM CPU.
Introduction: The Compilation Gap in Mixed-Precision Computing
To meet the massive computational demands of modern deep learning, hardware vendors have introduced specialized "tensorized instructions" such as Intel VNNI, NVIDIA Tensor Core, and ARM DOT. These instructions are optimized for mixed-precision arithmetic, typically multiplying low-precision inputs (e.g., int8 or fp16) and accumulating them into high-precision outputs (e.g., int32 or fp32) within a single cycle.
Despite their potential, these instructions are difficult to leverage. Standard compilers like GCC or LLVM struggle with the "horizontal accumulation" pattern required by these instructions. Without hardware-specific optimization, mixed-precision can actually degrade performance due to data casting overheads.
Currently, developers must choose between inflexible, "black-box" vendor libraries (like cuDNN or oneDNN) or error-prone manual assembly-level intrinsics. UNIT (Unifying Tensorized Instruction Compilation) bridges this gap by providing an automated framework to detect, map, and optimize these instructions across diverse hardware platforms.
Methodology: Unifying Hardware Semantics
UNIT is built upon the Apache TVM compiler stack and utilizes a three-stage approach to unify instruction compilation:
- Unified Semantics Abstraction: UNIT uses a Tensor Domain-Specific Language (DSL) to describe hardware instructions as small loop nests. This abstracts away chip-specific details into a common mathematical representation.
- Applicability Detection (The Inspector): This stage analyzes high-level operations (e.g., convolutions) to find "arithmetic isomorphism"—proving the model's math matches the hardware's capabilities.
- Rewriting and Tuning: Once a match is confirmed, the framework tiles and reorders loops to fit the hardware instruction perfectly. An automated tuner then explores the optimization space for threading and cache reuse.
The core innovation lies in treating hardware instructions as tiny programs, allowing the compiler to search for mathematical equivalence (isomorphism) rather than relying on hard-coded rules for every new chip.
Evaluation and Results
The researchers evaluated UNIT across standard benchmarks (ResNet, Inception, MobileNet) and 148 distinct convolution layers on three major platforms: Intel x86 (Cascade Lake), NVIDIA GPU (Tesla V100), and ARM CPU (Graviton2).
UNIT consistently outperformed industry-standard vendor libraries:
- Intel x86: 1.3× speedup over Intel oneDNN.
- NVIDIA GPU: 1.75× average speedup (up to 2.2×) over NVIDIA cuDNN.
- ARM CPU: 1.13× speedup over a manually tuned TVM implementation.
Furthermore, UNIT demonstrated impressive extensibility. By mapping 3D convolution math to 1D hardware instructions without core logic changes, it outperformed oneDNN by 1.2× on 3D workloads.
Conclusion and Future Work
UNIT successfully decouples the logic of hardware instructions from the performance optimization of the surrounding code. While it currently requires manual definition for operand packing rules on new platforms and faces challenges with highly irregular data access patterns, it represents a significant step toward a truly unified compiler for heterogeneous deep learning hardware. Future work aims to expand this framework to sparse operations and more specialized accelerators.
Summary updated