HARP: A Taxonomy for Heterogeneous and Hierarchical Processors for Mixed-reuse Workloads

Raveesh Garg, Michael Pellauer, Tushar KrishnaarXiv 2025 · DOI · arXiv

Introduction

In the early era of deep learning, hardware acceleration was primarily focused on dense Convolutional Neural Networks (CNNs). These workloads were characterized by high arithmetic intensity, performing massive amounts of computation on relatively small data footprints. However, the modern AI landscape has shifted toward Transformers (e.g., GPT, Llama) and Graph Neural Networks (GNNs). These "mixed-reuse" workloads present a significant challenge: they combine heavy matrix multiplications (GEMMs) with lightweight, bandwidth-hungry operations like Layer Normalization or autoregressive decoding.

Standard homogeneous accelerators, where every compute unit is identical, struggle to balance these competing demands. Optimizing for heavy math often leads to bandwidth starvation for lightweight tasks, while optimizing for bandwidth results in wasted silicon area for compute-heavy tasks. To address this, designers are increasingly turning to Hierarchical and Heterogeneous Processors (HHPs). The HARP framework provides a systematic taxonomy and evaluation tool to navigate this complex design space.

Comparison of roofline models for homogeneous vs. heterogeneous accelerators, highlighting the mismatch in arithmetic intensity for mixed-reuse workloads.

The HARP Taxonomy

The HARP framework classifies accelerators along two primary axes to provide a common language for architectural comparison:

  1. Compute Location: This axis distinguishes whether compute units are located only at the "leaves" (closest to processing elements, like a traditional NPU) or are distributed "hierarchically" throughout the memory hierarchy (e.g., Processing-in-Memory).
  2. Heterogeneity Source: This axis identifies where the architectural differences lie. It categorizes designs as Intra-node (different units sharing a controller), Cross-node (different units at the same memory level), or Cross-depth (different units at different memory levels, such as an NPU paired with logic-in-DRAM).

Evaluation Methodology

To evaluate these architectures, the researchers developed a tool wrapped around Timeloop, an industry-standard hardware modeling framework. This allows for the simulation of how mixed-reuse workloads are partitioned across different sub-accelerators. The study specifically focused on two partitioning strategies for Transformer models:

  • Intra-cascade partitioning: Overlapping operations within a single layer (e.g., overlapping attention score calculation with value generation).
  • Inter-cascade partitioning: Overlapping distinct phases of execution, such as the "prefill" phase of a new batch with the "decode" phase of a previous batch.
Intra- and inter-cascade partitioning strategies for Transformer workloads, showing how high-reuse and low-reuse operations are overlapped to improve throughput.

Key Insights and Results

The researchers evaluated HARP using representative models including BERT-large (encoder-only) and Llama-2/GPT-3 (decoder-only). Several critical insights emerged:

The Complementary Strategy

The primary advantage of HHPs is the ability to run high-reuse and low-reuse operations simultaneously. High-reuse tasks require significant math units but minimal bandwidth, while low-reuse tasks require high bandwidth but few math units. By pairing these tasks, HHPs can saturate chip resources more effectively than homogeneous designs.

Performance and Energy Efficiency

  • Decoder Dominance: For decoder-only models like Llama-2, heterogeneous accelerators significantly outperformed homogeneous ones by effectively decoupling the prefill and decode stages.
  • Energy Savings: "Cross-depth" architectures—those placing compute units near or inside DRAM—showed the highest efficiency, reducing energy consumption by 10% to 20% by minimizing data movement across the chip.
  • The Bandwidth Bottleneck: Performance is extremely sensitive to bandwidth partitioning. The authors found that a naive 50/50 split often erodes the benefits of heterogeneity, requiring careful tuning of resource allocation.

Conclusion and Future Work

While HHPs offer clear benefits for generative AI, they are not a universal solution. For encoder-only models like BERT, homogeneous designs can still prevail due to tight operational dependencies that limit overlapping opportunities. Furthermore, designers must avoid the risk of "dark silicon," where specialized sub-accelerators sit idle if the workload lacks sufficient task diversity.

Future work will involve applying the HARP taxonomy to emerging fields like neurosymbolic AI and AR/VR workloads, where operational diversity is even more pronounced. The framework also points toward several "derivable configurations"—novel chip architectures identified by the taxonomy that have yet to be physically realized.

Summary updated

Abstract

Artificial intelligence (AI) application domains consist of a mix of tensor operations with high and low arithmetic intensities (aka reuse). Hierarchical (i.e. compute along multiple levels of memory hierarchy) and heterogeneous (multiple different sub-accelerators) accelerators are emerging as a popular way to process mixed reuse workloads, and workloads which consist of tensor operators with diverse shapes. However, the space of hierarchical and/or heterogeneous processors (HHP's) is relatively under-explored. Prior works have proposed custom architectures to take advantage of heterogeneity to have multiple sub-accelerators that are efficient for different operator shapes. In this work, we propose HARP, a taxonomy to classify various hierarchical and heterogeneous accelerators and use the it to study the impact of heterogeneity at various levels in the architecture. HARP taxonomy captures various ways in which HHP's can be conceived, ranging from B100 cores with an "intra-node heterogeneity" between SM and tensor core to NeuPIM with cross-depth heterogeneity which occurs at different levels of memory hierarchy. We use Timeloop mapper to find the best mapping for sub-accelerators and also modify the Timeloop cost model to extend it to model hierarchical and heterogeneous accelerators.
BibTeX
@misc{garg2025harp,
  author = {Raveesh Garg and Michael Pellauer and Tushar Krishna},
  title = {{HARP: A Taxonomy for Heterogeneous and Hierarchical Processors for Mixed-reuse Workloads}},
  year = {2025},
  eprint = {2502.13113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13113},
}
Figure