Introduction
Deploying Large Language Models (LLMs) on edge devices—such as smartphones, IoT sensors, and local embedded systems—presents a significant hardware challenge. LLM inference is characterized by two distinct operational phases: the prefill stage and the decode stage. The prefill stage is compute-intensive, involving Matrix-Matrix Multiplications (GEMM) that benefit from high data reuse. Conversely, the decode stage, which generates tokens sequentially, is memory-bound due to Matrix-Vector Multiplications (GEMV), particularly at the batch size of one typical for edge applications.
Conventional hardware accelerators often struggle to maintain efficiency across both phases. Standard Systolic Arrays (SA) excel at GEMM but suffer from low utilization during the decode stage. Vector units offer flexibility for decoding but lack the data reuse capabilities required for efficient prefill. To bridge this gap, researchers at Cornell have developed the Hybrid Systolic Array (HSA), an accelerator designed to optimize both compute efficiency and memory utilization through hardware-software co-design.
Methodology
The HSA architecture employs a three-pronged approach to optimize edge LLM inference:
1. Hybrid Systolic Array (HSA) Architecture
The core of the design consists of 256 Processing Elements (PEs) organized into four clusters. The innovation lies in its reconfigurable dataflow: the hardware can function as a standard 2D systolic array during the prefill stage to maximize data reuse, and then switch to acting as independent vector units during the decode stage. This flexibility ensures 100% hardware utilization even during memory-bound GEMV operations.
2. MXINT4 Quantization and Shifting-Based Scaling
To minimize external memory access—the primary driver of energy consumption—the team implemented a hardware-friendly MXINT4 quantization scheme. Unlike traditional methods that require complex floating-point scaling factors, this approach uses power-of-two shifting-based scaling. This allows the hardware to perform dequantization using simple bit-shifters rather than area-intensive multipliers, reducing the dequantization logic area by approximately 10x compared to high-precision alternatives.
3. Optimized Non-linear Units
Beyond matrix operations, the researchers optimized common LLM bottlenecks:
- Fused RMSNorm: By fusing the normalization layer with the subsequent matrix multiplication, the design eliminates the need for a 32kB intermediate buffer, reducing latency by 5–10%.
- On-the-fly RoPE: Instead of utilizing large lookup tables for Rotary Positional Embeddings (RoPE), the accelerator calculates sine and cosine values online using trigonometric identities, further conserving memory bandwidth.
Evaluation and Results
The HSA accelerator was prototyped using TSMC 28nm CMOS technology and evaluated against several models, including RetNet (1.3B and 3B) and the Llama family (Llama 2, 3.1, and 3.2). RetNet was a primary focus due to its O(1) memory complexity during decoding, which aligns well with edge constraints.
Performance Metrics
- Area Efficiency: The design achieved 247 token/s/mm² in long-input scenarios, representing a 2.45× to 13.5× improvement over existing ASIC accelerators like MECLA or FACT.
- Power and Utilization: The chip operates at 0.108W at 500MHz and maintains 100% hardware utilization during the decode stage.
- Model Accuracy: Evaluation on WikiText2 and GSM8K confirmed that the 4-bit shift quantization maintains competitive perplexity (18.22 for RetNet-3B) and high generative accuracy (82.54% for Llama 3.1-8B), proving that hardware optimizations do not significantly degrade model performance.
Discussion and Limitations
A key insight from this research is the dominance of the decode stage in edge environments; even with long prompts, decoding can account for over 80% of total runtime. The HSA's ability to pivot between systolic and vector modes directly addresses this imbalance.
However, some limitations remain. The specialized units for RoPE and RMSNorm are tailored to specific mathematical formulations, meaning significant changes in LLM architectures might necessitate hardware revisions. Additionally, the current prototype is a small-scale 256-PE array. Future research will focus on scaling this hybrid architecture to support larger models (70B+ parameters) and adapting to evolving memory technologies like LPDDR6.
Conclusion
The Hybrid Systolic Array (HSA) demonstrates that through careful hardware-software co-design—specifically reconfigurable dataflows and hardware-aligned quantization—it is possible to achieve high-efficiency LLM inference on edge devices. By solving the "efficiency balance challenge" between prefill and decode stages, the HSA sets a new benchmark for area-efficient, low-power AI acceleration.
Summary updated