In the quest to make Transformer models fast enough and light enough to run on edge devices, deep learning hardware acceleration has historically focused on a single, massive target: matrix multiplication (GEMM). Because linear layers consume the vast majority of a model's parameters and raw floating-point operations (FLOPs), optimizing them yields immediate rewards. Through low-bit quantization (such as INT8, INT4, or FP8) and specialized hardware like Tensor Cores, we have become incredibly efficient at executing these linear layers.
However, this success has triggered Amdahl’s Law. As the computational cost of matrix multiplication drops toward zero, the relative cost of everything else skyrockets.
In modern, highly optimized Transformer pipelines, the primary performance bottleneck has shifted away from matrix multiplications and toward non-linear operations—specifically activation functions (GELU, SiLU) and normalization layers (Softmax, LayerNorm, RMSNorm). As Zhixiong Zhao et al. observe in QUARK, these non-linearities can account for up to half of total inference latency in low-precision models. Jiarui Fang et al. similarly note in TurboTransformers that auxiliary operations consume nearly 40% of execution time in standard frameworks, creating a severe performance imbalance.
Why Non-Linear Operations Resist Acceleration
To understand why these layers are so troublesome, we have to look at how hardware processes numbers. Specialized accelerators love Multiply-Accumulate (MAC) operations. They are predictable, uniform, and easily parallelized in hardware grids.
Non-linear operations, by contrast, require complex transcendental math and dynamic data coordination:
- Transcendental Math: Functions like Softmax and GELU require exponentiation ($e^x$), while LayerNorm and RMSNorm require square roots ($\sqrt{x}$) and divisions. Implementing high-precision floating-point units for these functions consumes an excessive amount of silicon area and power.
- Multi-Pass Data Movement: Softmax requires finding a maximum value across a vector, computing exponents, summing those exponents, and finally dividing each exponent by the sum. As Gamze İslamoğlu et al. explain in ITA, this multi-pass requirement forces the hardware to repeatedly move data back and forth from memory, consuming far more energy than the actual arithmetic.
- The Quantization Trap: Because these functions are highly sensitive to numerical precision, standard quantization frameworks often "cheat" by casting the integers back to high-precision floating-point (FP32 or FP16) just to compute the non-linear layer, and then quantizing the result back to integers. This constant type-casting destroys the throughput gains of the surrounding quantized linear layers.
To overcome this bottleneck, researchers are attacking the problem from four distinct angles: integer-only pipelines, look-up tables, hardware-software co-design, and mathematical elimination.
Strategy 1: Integer-Only Execution Pipelines
The most direct way to eliminate the type-casting overhead is to force non-linear operations to work entirely within integer arithmetic, never reverting to floating-point math.
Zejian Liu et al. pioneered this with FQ-BERT, a fully quantized BERT model that targets Softmax and LayerNorm directly, proving that full hardware acceleration is only possible when floating-point math is completely eradicated from the dataflow. In the sub-8-bit domain, Peiyan Dong et al. demonstrate in PackQViT that an integer-only dataflow is essential for achieving meaningful real-world speedups on mobile devices.
To make this practical on custom hardware, Alberto Marchisio et al. develop SwiftTron, an ASIC architecture that replaces power-hungry floating-point units with clever integer approximations of activation and normalization layers, drastically reducing the physical hardware footprint.
Strategy 2: Look-Up Tables and Piecewise Interpolation
If computing an exponent or a square root in hardware is too expensive, why not precompute the answers and store them in memory? This is the core intuition behind Look-Up Tables (LUTs) and interpolation.
Instead of relying on manual mathematical approximations, Joonsang Yu et al. introduce NN-LUT, a method that trains a tiny, single-hidden-layer neural network to mimic the non-linear function, which is then mathematically compiled into a highly efficient LUT. For quantized environments, Pingcheng Dong et al. propose GQA-LUT, utilizing a genetic algorithm to optimize quantization-aware LUTs that minimize silicon area and power consumption.
Standard Approach:
[INT8 Input] ──> Cast to FP32 ──> Expensive Math (e^x, √x) ──> Quantize ──> [INT8 Output]
LUT / Interpolation Approach:
[INT8 Input] ──> Index into LUT / Piecewise Linear Segment ───────────────> [INT8 Output]
When memory constraints prevent storing massive tables, piecewise linear interpolation offers a middle ground. JiangyongYu et al. introduce NLI, which uses non-uniform linear interpolation to approximate Softmax, SiLU, and RMSNorm. By breaking the curves of these functions into strategically sized linear segments, they achieve high-fidelity inference without high-precision floating-point hardware.
Strategy 3: Hardware-Software Co-Design and ISA Extensions
Sometimes, the cleanest solution requires modifying the underlying hardware instruction set architecture (ISA) to support non-linear math natively but cheaply.
Run Wang et al. introduce VEXP, a lightweight RISC-V ISA extension designed specifically to accelerate Bfloat16 exponentiation. By adding a tiny, specialized hardware unit that handles the core math of Softmax, they achieve a massive reduction in latency with a negligible 1% increase in total silicon area.
In the domain of In-Memory Computing, Mariam Rakka et al. present SoftmAP, a co-design that pairs polynomial approximations of Softmax with associative processors to keep computations entirely within memory arrays.
To maximize hardware efficiency without adding single-purpose silicon, the single-author paper by Jiajun Wu, TATAA, takes a dynamic approach: a transformable arithmetic architecture that dynamically repurposes the exact same integer processing units to handle both standard INT8 matrix multiplications and high-precision non-linear operations when needed.
Strategy 4: Eliminating Non-Linearity via Matrix Approximations
The most radical approach to solving the non-linear bottleneck is to eliminate the non-linear functions entirely. If a model can be reformulated to use only standard matrix arithmetic, it can leverage existing GEMM accelerators perfectly.
Neo Wei Ming et al. achieve this with MA-BERT. They replace Softmax, GELU, and LayerNorm entirely with matrix-arithmetic-friendly approximations. To prevent the model from losing accuracy due to this structural change, the authors use knowledge distillation to train the matrix-only model using a standard Transformer as a teacher. The result is a model that runs significantly faster on standard hardware because it never encounters a transcendental function.
Summary
The optimization of Transformers has reached a maturity point where optimizing matrix multiplication yields diminishing returns. By treating non-linear operations not as minor statistical adjustments, but as primary computational bottlenecks, this research direction ensures that the next generation of hardware accelerators can achieve true end-to-end efficiency.
Key Foundational Work
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss (2019) — contributed 1 concept elements to this direction.
Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices (2017) — contributed 1 concept elements to this direction.