Hardware Efficient Approximation of Non Linear Operations

232 concepts153 papers1 proposal

When deploying deep neural networks to edge hardware like FPGAs, ASICs, or microcontrollers, matrix multiplications are no longer the sole computational bottleneck. Quantization techniques can readily compress weights and linear layers into low-precision integers (such as INT8). However, a critical roadblock emerges at the boundaries of these layers: non-linear activation and normalization functions.

Functions like Softmax, GELU, LayerNorm, Sigmoid, and Tanh rely heavily on transcendental operations—such as exponentials ($e^x$), divisions ($1/x$), and inverse square roots ($1/\sqrt{x}$). On resource-constrained hardware lacking dedicated floating-point units (FPUs), computing these functions directly is incredibly expensive in terms of silicon area, latency, and power.

The Intuition Behind Approximation

Fortunately, deep neural networks exhibit a high degree of error resilience. They do not require infinite mathematical precision to maintain high task accuracy. This insight allows researchers to replace exact transcendental calculations with hardware-friendly approximations. By mapping complex curves to simple arithmetic operations—like bit-shifts, additions, and low-precision multiplications—we can bypass FPUs entirely.

Piecewise Linear and Affine Segmentation

The most intuitive way to approximate a complex curve is to chop it into smaller, straight-line segments. This is known as piecewise linear (PWL) approximation. Instead of calculating a continuous transcendental function, the hardware uses conditional logic to determine which segment an input falls into, and then evaluates a simple linear equation:

$$y = ax + b$$

To make this incredibly fast on hardware, researchers optimize the slope $a$ to be a power of two ($2^k$). In digital logic, multiplying a binary number by a power of two requires no multipliers at all; it is executed instantly via a bit-shift operation.

As shown by Aditya Kusupati et al. in FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network and Caiwen Ding et al. in C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs, multi-segment PWL approximations can replace Sigmoid and Tanh functions entirely. This enables integer-only execution on low-power microcontrollers.

Similarly, Jiyong Kim et al. apply fixed-segment PWL approximations in eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing to handle complex activations like SiLU on edge devices. Atli Kosson et al. extend this concept in Multiplication-Free Transformer Training via Piecewise Affine Operations to eliminate multiplications entirely across Transformer layers by using piecewise affine operations for exponential and logarithmic functions.

Look-Up Tables and Hybrid Correction

Another classic hardware strategy is the Look-Up Table (LUT), which precomputes function outputs and stores them in memory. When an activation is needed, the hardware simply indexes into the table using the input value. Yuan Liao 0004 et al. utilize LUTs within the Torch2Chip: An End-to-end Customizable Deep Neural Network Compression and Deployment Toolkit for Prototype Hardware Accelerator Design framework to deploy Vision Transformers using integer-only logic. Meanwhile, Joonsang Yu et al. demonstrate in NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference that a small neural network can be mathematically transformed directly into an optimized LUT, achieving massive power reductions for Softmax and LayerNorm.

However, high-precision LUTs can become prohibitively large, consuming valuable chip area. To solve this, researchers use hybrid architectures that combine simple functions with small tables. For example, Hanxue Liang et al. introduce a method in Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts where a standard, hardware-native ReLU function serves as a baseline approximation for GELU. A tiny LUT is then used to store only the residual difference (the calibration factor) between ReLU and GELU. This hybrid approach reduces hardware logic requirements by 90% while maintaining near-perfect mathematical fidelity.

Polynomials, Rational Functions, and Range Reduction

For hardware that possesses basic integer multipliers but lacks division or square-root logic, polynomial and rational function approximations offer an elegant alternative.

To make polynomial approximations highly accurate, researchers often employ a technique called range reduction. As noted by Qijun Huang et al. in Hardware Implementation of Single Iterated Multiplicative Inverse Square Root, transforming input values into a narrow range where the target function is only "softly non-linear" allows simple low-order polynomials to achieve extreme precision.

Gamze İslamoğlu et al. leverage this in ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers, executing Softmax entirely in the integer domain by combining base-2 exponentiation with bit-shifts to achieve a mean absolute error of less than 0.5%.

When dealing with non-traditional hardware, such as analog or memristor-based accelerators, mapping non-linearities is exceptionally challenging because crossbar arrays inherently perform linear matrix operations. In DS-LLM: Leveraging Dynamical Systems to Enhance Both Training and Inference of Large Language Models, Chuan Liu demonstrates that low-order Taylor expansions can map functions like Softmax and GELU onto analog hardware using native polynomial circuits, balancing the trade-off between circuit area and mathematical precision. Similarly, Abdul Rahoof uses Taylor series expansions in CapsBeam: Accelerating Capsule Network-Based Beamformer for Ultrasound Nonsteered Plane-Wave Imaging on Field-Programmable Gate Array to simplify exponential operations for Capsule Network routing on FPGAs.

Alternatively, Qi Liu et al. in Knowledge distillation for In-memory keyword spotting model and Reza Arablouei et al. in In-situ animal behavior classification using knowledge distillation and fixed-point quantization show that rational functions (ratios of polynomials, often derived via continued fractions) allow hardware with limited instruction sets to compute complex logarithmic and sigmoidal operations using only basic arithmetic.

Stochastic Computing and Finite State Machines

A more radical departure from traditional binary arithmetic is Stochastic Computing (SC), where numbers are encoded as the probability of finding a '1' in a random stream of bits. In this paradigm, complex multiplications collapse into a single, hardware-cheap AND gate.

To approximate non-linear functions like Tanh on these bit-streams, researchers replace arithmetic circuits entirely with Finite State Machines (FSMs). As shown by Ji Li et al. in SC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing, an FSM can track state transitions such that its steady-state output probability perfectly mirrors a hyperbolic tangent curve.

Xincheng Feng et al. expand on this with the SMURF architecture in Stochastic Multivariate Universal-Radix Finite-State Machine: a Theoretically and Practically Elegant Nonlinear Function Approximator, demonstrating that a unified FSM can approximate a wide variety of non-linear functions simply by adjusting internal sampling gate coefficients, drastically reducing area and power compared to traditional Taylor-series circuits.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 150 papers
  2. Version 1 · 2026-10-04 · 180 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_activation_function_approximation_85c703c8,
  title = {activation function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-function-approximation/rdr_85c703c81b994b0592e9835c26a39b0b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_activation_function_approximation_85c703c8_v2,
  title = {activation function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-function-approximation/rdr_85c703c81b994b0592e9835c26a39b0b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake