ReGate: Enabling Power Gating in Neural Processing Units
Introduction
The rapid expansion of AI workloads has made the energy consumption of data centers a significant sustainability concern. While Neural Processing Units (NPUs), such as Google’s TPUs, are optimized for high-throughput tensor computations, they often lack sophisticated power management. Research indicates that between 30% and 72% of an NPU's total energy is consumed as static power—leakage energy that persists even when hardware components are idle.
Traditional power-gating techniques used in CPUs and GPUs are difficult to apply to NPUs due to their unique architectural characteristics: massive systolic arrays with significant wake-up latencies, in-order execution models, and software-managed scratchpad SRAM. ReGate addresses these challenges through a hardware-software co-design that enables fine-grained, cycle-level power gating.
The ReGate Architecture
ReGate introduces a hybrid power management scheme tailored to the specific behavior of NPU components. The architecture focuses on minimizing static power across the systolic array, vector units, and memory systems.
The system categorizes components into hardware-managed and software-managed units:
- Systolic Arrays (Hardware-Managed): ReGate enables power gating at the individual Processing Element (PE) level. It leverages the deterministic "diagonal" dataflow inherent in systolic arrays to propagate wake-up signals just-in-time, effectively hiding wake-up latency.
- Interconnects & Memory Controllers (Hardware-Managed): Components like High Bandwidth Memory (HBM) controllers and inter-chip links (ICI) use lightweight hardware idle-detection to enter low-power modes during predictable idle periods.
- Vector Units & SRAM (Software-Managed): Due to highly variable idle patterns, these units are managed via the compiler. ReGate extends the NPU’s Instruction Set Architecture (ISA) with a
setpm(set power mode) instruction, allowing the compiler to explicitly power down idle units or reduce active SRAM capacity based on static program analysis.
Evaluation and Key Results
The effectiveness of ReGate was evaluated using a production-level NPU simulator validated against Google TPUv4 hardware. The test suite included diverse modern workloads such as Large Language Model (LLM) training and inference (Llama 2 and 3), recommendation models (DLRM), and generative AI (Stable Diffusion).
The results demonstrate significant energy savings across all models. On average, ReGate-Full reduced total chip energy by 15.5%, with peak savings reaching 32.8% in recommendation models characterized by high component idleness. Crucially, these savings were achieved with negligible performance impact (less than 0.5% degradation) thanks to the "just-in-time" wake-up logic.
Conclusion
ReGate proves that architectural idleness in NPUs is a significant opportunity for energy optimization. By combining hardware-driven diagonal wake-up for systolic arrays with compiler-driven management for memory and vector units, ReGate achieves substantial power reductions with minimal area overhead (3.3%). This approach not only improves the sustainability of AI infrastructure but also extends the effective operational lifespan of existing hardware. Future work will explore the integration of ReGate with dynamic voltage and frequency scaling (DVFS) and address potential security implications related to power-based side-channel attacks.
Summary updated