Abstract
Modern AI accelerators rely on Tensor Computing Units (TCUs) to handle the massive matrix multiplications required by Large Language Models (LLMs) and computer vision tasks. Most TCUs utilize Modified Booth Encoding (MBE) within each Processing Element (PE) to accelerate these operations. However, this decentralized approach leads to significant logic redundancy and area overhead. This paper presents EN-TensorCore (EN-T), a novel methodology that externalizes encoding logic and introduces a "wire-friendly" mathematical encoding scheme to minimize interconnect overhead. Experimental results using SMIC 40nm technology demonstrate that EN-T improves energy efficiency by up to 17.5% and area efficiency by 12.2% across diverse microarchitectures, effectively addressing critical scaling bottlenecks in AI hardware.
Introduction
In the landscape of AI hardware, the TCU is the primary engine for performance. To achieve high throughput, these units often employ Modified Booth Encoding (MBE) to reduce the number of partial products in multiplication. However, current architectures are inherently inefficient: every single PE in a massive multiplier array contains its own redundant encoding logic. Since the same multiplicands are frequently broadcast across multiple PEs, this logic is recalculated thousands of times, wasting precious die area and power.
As illustrated in Figure 1, while raw performance has increased over time, the efficiency gains in terms of TOPS/W and TOPS/MM^2 have faced diminishing returns. Furthermore, the area breakdown of modern chips shows that logic and interconnects occupy a substantial portion of the die. While externalizing the encoder seems like an obvious solution, standard MBE expands an $n$-bit input into $1.5n$ control bits. Moving this "fat" data across the chip creates massive wiring congestion, which often negates any area savings from logic removal.
Methodology
The proposed EN-T methodology overcomes these challenges through two core innovations: logic externalization and a novel mathematical encoding scheme.
Logic Externalization
The EN-T approach strips the encoder logic from individual PEs, leaving only the essential partial product compressors and adders. A centralized set of encoders is then placed at the edge of the array to serve multiple PEs simultaneously.
Wire-Friendly Encoding Polynomials
To solve the interconnect bottleneck, the authors developed a new mathematical encoding scheme. Unlike standard MBE, which requires 3 bits to represent 2 bits of input, the EN-T method encodes an $n$-bit number into just $n+1$ bits. By using mathematical induction, the team proved that any $n$-bit unsigned number can be decomposed into a series of 2-bit coefficients $\in \{-1, 0, 1, 2\}$. This representation is "wire-friendly," allowing encoded data to be distributed across the chip with minimal physical overhead, as shown in the comparison between traditional and EN-TensorCore microarchitectures in Figure 2.
Evaluation and Results
The EN-T architecture was implemented in RTL using SMIC 40nm technology and evaluated across five major microarchitectures: 2D Matrix, 1D/2D Array, Systolic Array (Weight Stationary and Output Stationary), and 3D Cube. The designs were tested against standard industry libraries and traditional MBE implementations using workloads such as ResNet, Inception V3, and VGG.
Key Performance Metrics
The methodology yielded significant improvements across computational scales ranging from 256 GOPS to 4 TOPS:
- Area Efficiency: Improved by an average of 8.7% to 12.2%.
- Energy Efficiency: Improved by an average of 13.0% to 17.5%.
- SoC Level Performance: During full neural network inference (e.g., ResNet50), total SoC power consumption dropped by 10% to 16%.
Architecture Sensitivity
The benefits of EN-T vary by architecture. The 1D/2D Array (similar to Cambricon designs) saw the highest gains, including a 20.2% increase in area efficiency at the 1 TOPS scale, due to its sensitivity to encoder logic overhead. In contrast, 3D Cube architectures (used by NVIDIA or Huawei Ascend) showed smaller improvements because their 3D structure already facilitates more efficient encoder sharing.
Conclusion
EN-TensorCore provides a scalable solution for enhancing the efficiency of AI accelerators by targeting redundant logic and interconnect overhead. While the new encoding method introduces a recursive carry-chain calculation, the resulting latency is negligible as the encoders are externalized and their outputs are registered. Future work will extend this methodology to floating-point formats (FP16, BF16) and validate the interconnect savings on more advanced process nodes like 5nm and 3nm.
Summary updated