Multi-Level Intermediate Representations for Hardware Acceleration

61 concepts49 papers2 proposals

Traditional software compilers are built like an hourglass. They take high-level source code written in languages like Python, C++, or domain-specific frameworks, and immediately flatten it into a single, uniform Intermediate Representation (IR)—such as standard Static Single Assignment (SSA) form or LLVM IR. From there, the compiler optimizes this flat sea of instructions before passing it to a backend that emits machine code.

While this hourglass model works exceptionally well for general-purpose CPUs, it breaks down completely when targeting specialized hardware accelerators like FPGAs, ASICs, TPUs, and custom RISC-V extensions.

The core issue is semantic loss. When a compiler takes a high-level operation—such as a matrix multiplication, a stencil loop, or a cryptographic gate—and immediately flattens it into scalar operations and conditional branches, the structural intent of the program is destroyed. The backend compiler cannot easily look at thousands of low-level instructions and deduce that they represent a clean, data-parallel operation. As a result, it misses critical macro-optimizations like loop tiling, memory partitioning, and task fusion.

To bridge this gap, a major research direction has emerged: Multi-Level Intermediate Representations. Instead of forcing a program into a single flat IR, modern hardware compilers use a tiered architecture of specialized dialects. By progressively lowering code through multiple levels of abstraction, these compilers preserve semantic intent exactly where it is needed, unlocking orders-of-magnitude performance gains.


The Core Mechanism: Progressive Lowering

The fundamental insight behind multi-level IRs is that different optimizations require different amounts of context.

Instead of performing a massive, single-step translation from high-level code to low-level hardware descriptions, modern compilers use progressive lowering. The program travels down a ladder of abstractions, stopping at distinct IR "dialects" along the way:

  1. High-Level Dialects: These capture domain-specific mathematics and algorithmic logic (e.g., tensor graphs or relational operations). At this level, the compiler performs hardware-independent optimizations, such as algebraic simplifications or operation reordering.
  2. Mid-Level Dialects: These capture structural execution patterns, such as loops, memory buffers, and dataflow streams. Here, the compiler optimizes memory layouts, hoists buffers, and schedules parallel tasks.
  3. Low-Level Dialects: These capture physical hardware constraints, such as bit-widths, register allocations, clock-cycle scheduling, and gate arrangements.

As noted in a comprehensive survey on The Deep Learning Compiler by Mingzhen Li et al., this multi-level structure allows high-level IRs to facilitate hardware-independent algebraic simplifications, while low-level IRs manage fine-grained loop transformations and memory access patterns.

By restructuring code before it is lowered to low-level hardware descriptions, systematic passes achieve substantial speedups without requiring manual code annotations. For example, Nicolas Bohm Agostini et al. demonstrate in Bridging Python to Silicon: The SODA Toolchain that staged lowering allows the compiler to perform complex transformations, such as loop tiling and buffer hoisting, at the exact stage where data dependencies are most transparent.


Decoupling Logic from Structure

Why does this hierarchical approach work so much better than traditional compilation? Because it allows compilers to decouple functional logic from structural hardware constraints.

In a flat compiler, optimizing a dataflow pipeline is incredibly difficult because the functional task boundaries are tangled up with physical hardware details like memory ports and timing. Multi-level IRs solve this by separating these concerns into different layers.

[ High-Level Domain Logic ]  --> e.g., Tensor Algebra, Stencil Operations
            │
            ▼
[ Functional Dataflow IR ]   --> Global optimizations (Task Fusion, Splitting)
            │
            ▼
[ Structural Dataflow IR ]   --> Physical constraints (Memory Ports, Timing)
            │
            ▼
[ Low-Level Hardware IR ]    --> Bit-widths, Registers, Gate Netlists

In HIDA: A Hierarchical Dataflow Compiler for High-Level Synthesis, Deming Chen et al. introduce a two-level IR that explicitly separates functional dataflow from structural dataflow. At the functional level, tasks are treated as transparent nodes. The compiler can freely fuse or split tasks to optimize the global pipeline without worrying about clock cycles or physical wiring. Once the optimal boundaries are set, the compiler lowers the design to the structural level to handle physical memory ports and timing constraints.

Similarly, Hanchen Ye et al. show in ScaleHLS: Scalable High-Level Synthesis through MLIR that representing hardware designs across multiple levels of abstraction—from high-level computation graphs to low-level hardware directives—allows the compiler to automate complex design space exploration (DSE). Transformations like dataflow parallelism and memory partitioning are applied at their most effective scope, yielding performance gains several orders of magnitude higher than traditional single-level synthesis tools.


Preserving Intent Down to the Metal

The power of multi-level IRs extends all the way to the final hardware generation phase. Traditional compiler backends act as a bottleneck because they strip away high-level intent just when specialized hardware needs it most.

In A Multi-level Compiler Backend for Accelerated Micro-kernels Targeting RISC-V ISA Extensions, Anton Lydike demonstrates that a backend processing multiple levels of abstraction simultaneously can map high-level mathematical operations directly to specialized hardware instructions and memory-stream extensions. By keeping structural information like loops and data streams intact until the final emission phase, the compiler achieves near-peak performance without requiring hand-written assembly.

This philosophy applies equally well to hardware description languages themselves. Fabian Schuiki et al. introduce LLHD: a multi-level intermediate representation for hardware description languages, which uses a tiered IR architecture to span the entire design flow—from high-level behavioral simulation down to gate-level netlists. This allows a single format to represent complex control flow, hierarchical structural relations, and physical gate arrangements within a unified framework.


Entering the Pipeline: Staging and Raising

To leverage a multi-level IR, compilers must first ingest high-level code into their initial top-level dialects. Researchers achieve this using two primary paradigms:

Top-Down: Staged Programming

Staged programming treats a high-level language as a metaprogramming tool to construct an optimized execution graph. As Tiark Rompf et al. show in Building-Blocks for Performance Oriented DSLs, staged program generation constructs an IR graph during execution, allowing for complex analysis and transformation before final code emission.

Fei Wang 0046 et al. apply this concept to machine learning in A Language and Compiler View on Differentiable Programming, using multi-stage programming and type-level annotations to distinguish between code that should be executed immediately and code that should be transformed into an IR. This allows developers to use a simple, expressive library interface while the system automatically compiles the code into highly optimized hardware-specific kernels.

Bottom-Up: Program Synthesis and Raising

Instead of writing new code in specialized languages, what if we could automatically upgrade legacy code?

In mlirSynth: Automatic, Retargetable Program Raising in Multi-Level IR using Program Synthesis, Tobias Grosser demonstrates an automated search framework that "raises" low-level code (like standard C) back up into high-level domain-specific dialects. Because high-level IRs capture structural intent more effectively than low-level code, this raising process allows legacy applications to unlock hardware-specific optimizations on accelerators like TPUs without manual rewriting.


A Unified Solution for Diverse Domains

The multi-level IR paradigm is not a niche optimization for deep learning; it has become the standard architectural pattern for compiling complex computations across a wide variety of domains:

By replacing the traditional single-level "hourglass" compiler with a multi-level ladder of abstractions, modern compiler design provides a systematic, scalable way to map high-level human intent to the complex spatial realities of modern hardware accelerators.

Key Foundational Work

Compilation for explicitly managed memory hierarchies (2007) — contributed 2 concept elements to this direction.
Building-Blocks for Performance Oriented DSLs (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_level_intermediate_representation_ac1858db,
  title = {multi-level intermediate representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-level-intermediate-representation/rdr_ac1858dba78b450eb0c4e6323060a4a2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_level_intermediate_representation_ac1858db_v1,
  title = {multi-level intermediate representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-level-intermediate-representation/rdr_ac1858dba78b450eb0c4e6323060a4a2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake