Abstract
Vision Transformers (ViTs) achieve state-of-the-art performance in computer vision but face significant deployment challenges on edge hardware like FPGAs due to computationally expensive non-linear functions. PEANO-ViT introduces a suite of hardware-optimized, division-free approximations for Layer Normalization, Softmax, and GELU. By replacing transcendental functions with bit-manipulation, small lookup tables, and piece-wise linear segments, the proposed framework achieves up to an 8x improvement in power efficiency with negligible accuracy loss (<0.5%).
Introduction
The transition of Vision Transformers from high-performance servers to edge devices is hindered by "hardware-unfriendly" non-linear layers. Specifically, Layer Normalization (LN), Softmax, and GELU activations require complex operations such as reciprocals of square roots, exponentials, and transcendental functions. On FPGAs, these operations typically consume excessive resources (DSPs and LUTs) or introduce high latency through iterative methods. PEANO-ViT addresses these bottlenecks through hardware-software co-design, prioritizing digital logic strengths like bit-shifting and small memory lookups over raw floating-point arithmetic.
Proposed Approximation Strategies
The core of PEANO-ViT lies in three distinct strategies designed to eliminate expensive division and transcendental operations.
Division-Free Layer Normalization
Standard LN requires calculating $1/\sqrt{x}$. PEANO-ViT avoids direct calculation by using a log-based identity. The hardware identifies the leading '1' bit of the input to approximate the exponent, while a small lookup table (LUT) handles the mantissa. This approach transforms the operation into a series of shifts and additions.
Padé-based Softmax
Softmax implementation is optimized in two stages:
- Exponentials: A [2,2] Padé approximant is used instead of a Taylor series. This rational function provides a tighter fit for $e^x$ over the relevant range.
- Multi-Scale Reciprocal (MSR): To remove the normalization division, a dynamic scaling factor maps the denominator into a narrow range, allowing the use of a small pre-stored reciprocal table for multiplication.
Piece-wise Linear (PWL) GELU
The complex GELU formula is replaced by a 7-segment linear approximation. The breakpoints (e.g., at $x = -3, -2.1, -0.75, 0, 0.5, 3$) were optimized to minimize Mean Square Error (MSE), ensuring the approximation tracks the original function's S-curve and behavior near zero.
Experimental Results
The researchers evaluated PEANO-ViT on the ImageNet-1K dataset using architectures including DeiT-S, DeiT-B, Swin-B, and ViT-L. Implementation on a Xilinx UltraScale+ VU9P FPGA demonstrated significant gains across multiple metrics.
Accuracy and Resource Efficiency
The approximations result in minimal accuracy degradation; for instance, DeiT-B accuracy dropped by only 0.2% (from 81.85% to 81.65%). More importantly, resource usage was drastically reduced. For the GELU layer, LUT usage decreased by 97.1% and DSP usage by 87.5% compared to standard implementations.
Power and Throughput
Power efficiency improvements were substantial:
- Layer Norm: 1.91× improvement
- Softmax: 1.39× improvement
- GELU: 8.01× improvement
High throughput is maintained through a Level of Parallelism (LoP) of 16 and FIFO-based data flow management to minimize stalls.
Conclusion and Future Work
PEANO-ViT demonstrates that hardware-software co-design can effectively mitigate the computational costs of non-linearities in Transformers. While the method requires a brief two-epoch fine-tuning phase to recover accuracy, the resulting power and resource savings are transformative for edge deployment. Future work includes extending this framework to Large Language Models (LLMs), where similar non-linear bottlenecks exist at a larger scale.
Summary updated