EN-T: Optimizing Tensor Computing Engines Performance via Encoder-Based Methodology

Qizhe Wu, Yuchen Gui, Zhichen Zeng, Xiaotian Wang, Huawen Liang, Xi Jin

TL;DR

This paper introduces EN-T, a novel encoder-based methodology to optimize Tensor Computing Units (TCUs) by reducing redundant computations. By centralizing the encoding of multiplicands, EN-T significantly improves area and energy efficiency across various TCU architectures, offering a compatible and versatile solution for modern AI hardware.

What Problem Does This Paper Address?

Modern AI applications, from LLMs to self-driving cars, heavily rely on tensor computations, primarily matrix multiplications. This has driven the development of specialized hardware like NVIDIA's Tensor Cores and Google's TPUs. However, existing architectures often perform redundant computations within their processing elements (PEs), specifically in the encoding step of multiplication. This redundancy contributes to increased chip area and power consumption. The paper aims to address this by identifying and eliminating this overlooked computation reuse opportunity within TCUs.

Methods

The core of the EN-T methodology lies in rethinking the data flow for multiplication. Traditionally, each multiplier within a TCU's PEs performs its own encoding of the multiplicand. EN-T proposes to remove this per-PE encoder and instead introduce a single encoder for each column of the multiplicand pathway outside the array. This means that only the encoded multiplicand is broadcast to the PEs, rather than the original multiplicand which would then be encoded by each PE.

The paper also introduces a modified encoding polynomial that aims to reduce the bit width of the encoded numbers compared to standard methods like Modified Booth Encoding (MBE). This is achieved by restructuring coefficients and using a recursive method to derive the encoding. The EN-T architecture is designed to be compatible with existing TCU microarchitectures, including 2D Matrix, 1D/2D Array, Systolic Array (OS and WS), and 3D Cube.

Datasets

Not discussed in the paper. The evaluation uses standard AI workloads (ResNet34, ResNet50, ResNet101, Inception_V3, DenseNet121, DenseNet161, Vgg13, Vgg19) for System-on-Chip (SoC) level benchmarking.

Key Insights and Ideas

  • Computation Reuse is Key: The paper highlights that a significant opportunity for optimization in TCUs lies in reusing the encoded multiplicand, rather than re-encoding it in every PE.
  • Centralized Encoding: Moving the encoder outside the PE array and broadcasting the encoded data is a simple yet effective way to reduce redundancy.
  • Encoder Design Matters: The specific encoding polynomial and logic used can impact the overall efficiency. The proposed modified encoding aims for a more compact representation.
  • Architecture Compatibility: The EN-T approach is designed to be flexible and can be integrated into various existing TCU microarchitectures, making it a practical improvement.
  • Trade-offs Exist: While the proposed encoder might have a slightly higher area/delay in isolation, its integration into the EN-T architecture leads to overall system-level improvements.

Key Results

The EN-T architecture demonstrated significant improvements across various TCU microarchitectures and computational scales:

  • Area Efficiency: Average improvement of 8.7%, 12.2%, and 11.0% for computational scales of 256 GOPS, 1 TOPS, and 4 TOPS, respectively.
  • Energy Efficiency: Average enhancement of 13.0%, 17.5%, and 15.5% for the same computational scales.
  • Specific Architectures: The 1D/2D Array architecture showed a notable 20.2% increase in area efficiency and 20.5% in energy efficiency at 1 TOPS.
  • SoC Level: At the SoC level, energy consumption reduction ranged from 5.0% to 16.0% depending on the TCU architecture and workload.

Limitations and Future Work

  • Latency in Carry-Chain Encoding: The paper acknowledges that the proposed encoding method, based on carry-chain encoding, can lead to gradually increasing latency with the width of the multiplicand. However, they argue this is mitigated by placing encoders outside the array and using registers.
  • 3D Cube Architecture Benefits: The energy saving benefits for the 3D Cube architecture were lower compared to 2D architectures, attributed to the need for more encoders for the same computational power in a 3D structure.
  • Future Work: The paper suggests further exploration of encoder performance and its integration into large-scale TCUs. The experimental evaluation focused on INT8 precision, and future work could explore other precisions.

Summary updated

Abstract

Tensor computations, with matrix multiplication being the primary operation, serve as the fundamental basis for data analysis, physics, machine learning, and deep learning. As the scale and complexity of data continue to grow rapidly, the demand for tensor computations has also increased significantly. To meet this demand, several research institutions have started developing dedicated hardware for tensor computations. To further improve the computational performance of tensor process units, we have reexamined the issue of computation reuse that was previously overlooked in existing architectures. As a result, we propose a novel EN-T architecture that can reduce chip area and power consumption. Furthermore, our method is compatible with existing tensor processing units. We evaluated our method on prevalent microarchitectures, the results demonstrate an average improvement in area efficiency of 8.7 %, 12.2 %, and 11.0 % for tensor computing units at computational scales of 256 GOPS, 1 TOPS, and 4 TOPS, respectively. Similarly, there were energy efficiency enhancements of 13.0 %, 17.5 %, and 15.5 %.
BibTeX
@inproceedings{wu2024en,
  author = {Qizhe Wu and Yuchen Gui and Zhichen Zeng and Xiaotian Wang and Huawen Liang and Xi Jin},
  title = {{EN-T: Optimizing Tensor Computing Engines Performance via Encoder-Based Methodology}},
  booktitle = {ICCD},
  year = {2024},
  url = {https://openreview.net/forum?id=hSdMy5TYK6},
}
Figure