TL;DR
This paper introduces EN-T, a novel encoder-based methodology to optimize Tensor Computing Units (TCUs) by reducing redundant computations. By centralizing the encoding of multiplicands, EN-T significantly improves area and energy efficiency across various TCU architectures, offering a compatible and versatile solution for modern AI hardware.
What Problem Does This Paper Address?
Modern AI applications, from LLMs to self-driving cars, heavily rely on tensor computations, primarily matrix multiplications. This has driven the development of specialized hardware like NVIDIA's Tensor Cores and Google's TPUs. However, existing architectures often perform redundant computations within their processing elements (PEs), specifically in the encoding step of multiplication. This redundancy contributes to increased chip area and power consumption. The paper aims to address this by identifying and eliminating this overlooked computation reuse opportunity within TCUs.
Methods
The core of the EN-T methodology lies in rethinking the data flow for multiplication. Traditionally, each multiplier within a TCU's PEs performs its own encoding of the multiplicand. EN-T proposes to remove this per-PE encoder and instead introduce a single encoder for each column of the multiplicand pathway outside the array. This means that only the encoded multiplicand is broadcast to the PEs, rather than the original multiplicand which would then be encoded by each PE.
The paper also introduces a modified encoding polynomial that aims to reduce the bit width of the encoded numbers compared to standard methods like Modified Booth Encoding (MBE). This is achieved by restructuring coefficients and using a recursive method to derive the encoding. The EN-T architecture is designed to be compatible with existing TCU microarchitectures, including 2D Matrix, 1D/2D Array, Systolic Array (OS and WS), and 3D Cube.
Datasets
Not discussed in the paper. The evaluation uses standard AI workloads (ResNet34, ResNet50, ResNet101, Inception_V3, DenseNet121, DenseNet161, Vgg13, Vgg19) for System-on-Chip (SoC) level benchmarking.
Key Insights and Ideas
- Computation Reuse is Key: The paper highlights that a significant opportunity for optimization in TCUs lies in reusing the encoded multiplicand, rather than re-encoding it in every PE.
- Centralized Encoding: Moving the encoder outside the PE array and broadcasting the encoded data is a simple yet effective way to reduce redundancy.
- Encoder Design Matters: The specific encoding polynomial and logic used can impact the overall efficiency. The proposed modified encoding aims for a more compact representation.
- Architecture Compatibility: The EN-T approach is designed to be flexible and can be integrated into various existing TCU microarchitectures, making it a practical improvement.
- Trade-offs Exist: While the proposed encoder might have a slightly higher area/delay in isolation, its integration into the EN-T architecture leads to overall system-level improvements.
Key Results
The EN-T architecture demonstrated significant improvements across various TCU microarchitectures and computational scales:
- Area Efficiency: Average improvement of 8.7%, 12.2%, and 11.0% for computational scales of 256 GOPS, 1 TOPS, and 4 TOPS, respectively.
- Energy Efficiency: Average enhancement of 13.0%, 17.5%, and 15.5% for the same computational scales.
- Specific Architectures: The 1D/2D Array architecture showed a notable 20.2% increase in area efficiency and 20.5% in energy efficiency at 1 TOPS.
- SoC Level: At the SoC level, energy consumption reduction ranged from 5.0% to 16.0% depending on the TCU architecture and workload.
Limitations and Future Work
- Latency in Carry-Chain Encoding: The paper acknowledges that the proposed encoding method, based on carry-chain encoding, can lead to gradually increasing latency with the width of the multiplicand. However, they argue this is mitigated by placing encoders outside the array and using registers.
- 3D Cube Architecture Benefits: The energy saving benefits for the 3D Cube architecture were lower compared to 2D architectures, attributed to the need for more encoders for the same computational power in a 3D structure.
- Future Work: The paper suggests further exploration of encoder performance and its integration into large-scale TCUs. The experimental evaluation focused on INT8 precision, and future work could explore other precisions.
Summary updated