Retargetable Compilation for Tensor Kernels via a Unified Layout Abstraction

Likai Ma, Renze Chen, Tianqi Li, Yuhao Luo, Youwei Xiao, Size Zheng, Xiuhong Li, Yun LiangarXiv 2026 · arXiv

TL;DR

Arknife is a new retargetable compiler that treats tensor data layouts as first-class instruction requirements rather than mere performance hints. By using a unified layout algebra and a global constraint solver, it automatically generates high-performance kernels for diverse hardware—including x86 CPUs, NPUs, and GPUs—outperforming vendor-optimized libraries by up to 3.78x.

What Problem Does This Paper Address?

Modern hardware accelerators (like GPUs and NPUs) derive their massive performance from specialized instructions, such as NVIDIA's Tensor Cores or Qualcomm's HMX units. However, these instructions are incredibly "picky": they require data to be in very specific "layouts" in memory or registers to execute correctly and efficiently. For example, an NPU might require rows to be interleaved in pairs, while a GPU wants data distributed across parallel "lanes" in a complex, non-linear pattern.

Different hardware requires different data layouts for the same logical operation.

Currently, developers must manually manage these layouts or use compilers that are locked into a single hardware target. This makes porting code to new chips a manual, error-prone nightmare. Furthermore, if a compiler makes a layout choice locally that doesn't fit a later instruction, it must insert expensive "repair" operations (like data shuffles), which can be orders of magnitude slower than the actual computation.

Methods

The authors introduce Arknife, a compiler built on a unified layout algebra. Instead of hardcoding data movement, Arknife treats layouts as a set of constraints to be solved.

  1. Constraint-Based Algebra: It maps logical tensor axes (like "row" and "column") to hardware axes (like "register index" or "thread ID"). Instructions simply declare their requirements as partial mappings.
  2. Instruction Binding & Feasibility: The compiler selects target instructions and checks if their layout needs can be met. If there is a conflict, it inserts "provisional repairs" (e.g., a transpose or shuffle).
  3. Global Optimization: Arknife uses e-graph rewriting to find and eliminate redundant repairs. For instance, two back-to-back transposes might be simplified or canceled out.
  4. Global Solver: Finally, it uses an Integer Linear Programming (ILP) solver to find a single, consistent layout for every tensor in the program that satisfies all instruction requirements simultaneously across the entire compute graph.
Arknife solves layout constraints globally across the entire compute graph, unlike previous local or constructive approaches.

Datasets

The researchers evaluated Arknife on three distinct architectures using standard deep learning workloads:

  • x86 CPU: Intel Xeon with AVX-512 VNNI instructions.
  • Qualcomm Hexagon NPU: Utilizing HMX matrix instructions.
  • NVIDIA A100 GPU: Utilizing Tensor Core MMA instructions.

Workloads included GEMM (General Matrix Multiplication), Multi-Head Attention (MHA), LayerNorm, Softmax, and quantized operations. They compared Arknife against vendor libraries (oneDNN, QNN, cuBLAS) and state-of-the-art compilers like Halide, Hydride, MISAAL, and Triton.

Key Insights and Ideas

  • Layout is Semantic: The paper argues that layout is part of an instruction's functional interface. If the layout is wrong, the instruction might produce incorrect results or fail to run entirely.
  • Global vs. Local Solving: Most compilers fix layout mismatches locally. Arknife’s "aha moment" is that by solving for layouts globally, you can often choose an initial layout that makes all subsequent instructions "just work" without any expensive data movement.
  • Retargetability via Abstraction: Because the layout algebra is independent of any specific Instruction Set Architecture (ISA), adding support for a new chip only requires writing a short description of its instructions and their layout needs—the compiler's core logic remains unchanged.

Key Results

Arknife delivers significant speedups by drastically reducing data movement overhead:

  • x86 Performance: It achieves a 2.17x geometric mean speedup over Intel's oneDNN and 5.08x over the retargetable compiler MISAAL.
  • Hexagon NPU Performance: It outperforms the vendor library QNN by 3.78x and MISAAL by 4.53x.
  • NVIDIA A100 Performance: It reaches 97% of the performance of the highly optimized cuBLAS on TF32 GEMM and is 5.04x faster than unfused PyTorch for FP16 MHA.
  • Programming Efficiency: Arknife can implement complex kernels like MHA with just 68 lines of code, compared to 117–650 lines for manual implementations in other frameworks.
An example of Arknife compiling a quantized addition for the Hexagon NPU, simplifying repairs and solving for the optimal axis mapping.

Limitations and Future Work

  • Complex Domains: The current model assumes regular, rectangular tensor domains. It does not yet handle "ragged" tensors or data-dependent access patterns.
  • Weight Management: While it excels at kernel execution, the authors note that automatic weight preprocessing and managing where weights reside in memory (residency) are areas for future work.
  • Fusion Scope: The authors suggest expanding the scope of "megakernel" fusion to even larger operator graphs to further minimize global memory traffic.

Summary updated

Abstract

Modern hardware accelerators achieve high performance through specialized instructions coupled with tightly managed memory hierarchies. These instructions impose strict layout requirements on their operands: where tensor fragments reside, how they are organized, and how parallel units access them. Such instruction-visible layout is critical to kernel performance, as it determines whether instructions execute correctly and how much data movement they incur. However, existing compilers either scatter layout requirements across handwritten programs and local scheduling rules, or provide layout representations and conversions specific to a single class of targets. Both approaches expose the same abstraction gap: the absence of a unified model for instruction layout requirements across backends. This paper presents Arknife, a retargetable compiler that treats instruction-visible layout as first-class instruction semantics. Its central idea is a constraint-based layout algebra: tensor layouts map logical axes to hardware axes spanning parallel levels and memory spaces. Instruction operands declare layout requirements, and target-provided repairs resolve conflicts. Layouts are then solved globally across the program. ISA-specific layout facts are supplied by target-side instruction descriptions rather than hardcoded in the compiler. Retargeting reuses the layout algebra and solver while defining the new target's instructions and their operand requirements. We evaluate Arknife across x86 CPU, Hexagon NPU, and NVIDIA A100 GPU. Arknife matches cuBLAS on A100, and on x86 and Hexagon outperforms vendor-optimized libraries by $2.17\times$ and $3.78\times$ and the retargetable compiler MISAAL by $5.08\times$ and $4.53\times$, respectively.
BibTeX
@misc{ma2026retargetable,
  author = {Likai Ma and Renze Chen and Tianqi Li and Yuhao Luo and Youwei Xiao and Size Zheng and Xiuhong Li and Yun Liang},
  title = {{Retargetable Compilation for Tensor Kernels via a Unified Layout Abstraction}},
  year = {2026},
  eprint = {2611.13741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13741},
}

Loading related content...

Figure