Introduction
In the early era of deep learning, hardware acceleration was primarily focused on dense Convolutional Neural Networks (CNNs). These workloads were characterized by high arithmetic intensity, performing massive amounts of computation on relatively small data footprints. However, the modern AI landscape has shifted toward Transformers (e.g., GPT, Llama) and Graph Neural Networks (GNNs). These "mixed-reuse" workloads present a significant challenge: they combine heavy matrix multiplications (GEMMs) with lightweight, bandwidth-hungry operations like Layer Normalization or autoregressive decoding.
Standard homogeneous accelerators, where every compute unit is identical, struggle to balance these competing demands. Optimizing for heavy math often leads to bandwidth starvation for lightweight tasks, while optimizing for bandwidth results in wasted silicon area for compute-heavy tasks. To address this, designers are increasingly turning to Hierarchical and Heterogeneous Processors (HHPs). The HARP framework provides a systematic taxonomy and evaluation tool to navigate this complex design space.
The HARP Taxonomy
The HARP framework classifies accelerators along two primary axes to provide a common language for architectural comparison:
- Compute Location: This axis distinguishes whether compute units are located only at the "leaves" (closest to processing elements, like a traditional NPU) or are distributed "hierarchically" throughout the memory hierarchy (e.g., Processing-in-Memory).
- Heterogeneity Source: This axis identifies where the architectural differences lie. It categorizes designs as Intra-node (different units sharing a controller), Cross-node (different units at the same memory level), or Cross-depth (different units at different memory levels, such as an NPU paired with logic-in-DRAM).
Evaluation Methodology
To evaluate these architectures, the researchers developed a tool wrapped around Timeloop, an industry-standard hardware modeling framework. This allows for the simulation of how mixed-reuse workloads are partitioned across different sub-accelerators. The study specifically focused on two partitioning strategies for Transformer models:
- Intra-cascade partitioning: Overlapping operations within a single layer (e.g., overlapping attention score calculation with value generation).
- Inter-cascade partitioning: Overlapping distinct phases of execution, such as the "prefill" phase of a new batch with the "decode" phase of a previous batch.
Key Insights and Results
The researchers evaluated HARP using representative models including BERT-large (encoder-only) and Llama-2/GPT-3 (decoder-only). Several critical insights emerged:
The Complementary Strategy
The primary advantage of HHPs is the ability to run high-reuse and low-reuse operations simultaneously. High-reuse tasks require significant math units but minimal bandwidth, while low-reuse tasks require high bandwidth but few math units. By pairing these tasks, HHPs can saturate chip resources more effectively than homogeneous designs.
Performance and Energy Efficiency
- Decoder Dominance: For decoder-only models like Llama-2, heterogeneous accelerators significantly outperformed homogeneous ones by effectively decoupling the prefill and decode stages.
- Energy Savings: "Cross-depth" architectures—those placing compute units near or inside DRAM—showed the highest efficiency, reducing energy consumption by 10% to 20% by minimizing data movement across the chip.
- The Bandwidth Bottleneck: Performance is extremely sensitive to bandwidth partitioning. The authors found that a naive 50/50 split often erodes the benefits of heterogeneity, requiring careful tuning of resource allocation.
Conclusion and Future Work
While HHPs offer clear benefits for generative AI, they are not a universal solution. For encoder-only models like BERT, homogeneous designs can still prevail due to tight operational dependencies that limit overlapping opportunities. Furthermore, designers must avoid the risk of "dark silicon," where specialized sub-accelerators sit idle if the workload lacks sufficient task diversity.
Future work will involve applying the HARP taxonomy to emerging fields like neurosymbolic AI and AR/VR workloads, where operational diversity is even more pronounced. The framework also points toward several "derivable configurations"—novel chip architectures identified by the taxonomy that have yet to be physically realized.
Summary updated