For years, the deep learning community has treated quantization as a post-processing step. We train models using power-hungry, high-precision floating-point math (FP32 or FP16) on massive data centers, and then compress them into low-bit integers (like INT8) so they can run efficiently on edge devices.
But what if we could eliminate floating-point math from the entire lifecycle?
Executing the entire training loop—forward pass, backward pass, and weight updates—using only integer arithmetic has long been a holy grail for hardware efficiency. Doing so allows on-device learning on microcontrollers, FPGAs, and edge hardware that lack dedicated floating-point units (FPUs). Recent research demonstrates that integer-only training is not only possible but can match the convergence and accuracy of standard floating-point pipelines.
The Core Challenge: Why Gradients Break Naive Quantization
To understand why integer-only training is difficult, we have to look at how gradient descent works. During training, activations and weights can span vastly different ranges across layers. More importantly, the gradients computed during the backward pass are often orders of magnitude smaller than the weights themselves.
If you attempt to naively cast these values into a standard 8-bit integer format, two things happen:
- Underflow: Tiny gradient updates round down to zero, causing the model to stop learning entirely.
- Overflow: Large activation values or sudden gradient spikes exceed the maximum integer limit, corrupting the weight updates.
Floating-point numbers avoid this because they have a dynamic exponent that automatically scales with the magnitude of the number. Integers have a fixed spacing. To make integer-only training work, we must replicate this dynamic range without paying the steep hardware cost of floating-point arithmetic units.
Mechanism 1: Shared Exponents and Block Floating Point
The primary way to achieve floating-point flexibility with integer hardware is to decouple the exponent from individual numbers. Instead of giving every single value its own exponent (as in FP32), we can group a block of numbers—such as a row of a matrix or an entire tensor—and force them to share a single exponent. The underlying data is stored and multiplied as pure integers, while the shared exponent is managed separately in software or simple control logic.
Early foundations for this approach were laid by John Wawrzynek et al., who demonstrated that fixed-point arithmetic with 16-bit weights and 8-bit activations provided sufficient accuracy for backpropagation. Building on this concept, Dipankar Das et al. introduced a Dynamic Fixed Point (DFP) scheme that allowed large convolutional networks like ResNet-50 to be trained entirely with 16-bit integer operations by using shared exponents to manage the dynamic range of gradients.
More recently, this concept has been pushed to even lower bit-widths. Sifan Zhou et al. proposed a Group-Shared Exponents Integer (GSE-INT) format specifically for fine-tuning Large Language Models (LLMs) on-device, slashing memory usage by half compared to FP8. Similarly, Marcelo Gennari do Nascimento et al. utilized a generalized quantization scheme to optimize hardware multipliers, proving that 4-bit integer arithmetic can preserve a model's convergence properties throughout its training lifecycle.
Mechanism 2: Taming Activation Outliers
Even with shared exponents, certain neural network architectures present severe spatial distribution challenges. Transformers, for instance, are notorious for producing "activation outliers"—specific channels where values explode to magnitudes significantly higher than the rest of the tensor. If the shared exponent scales up to accommodate these rare outliers, the remaining normal values are squashed down and lose all precision.
To solve this, researchers have developed techniques to smooth out or isolate these outliers before quantization occurs:
- Distribution Smoothing: Haocheng Xi et al. introduced a method for training Transformers with 4-bit integers by using a Hadamard quantizer. The Hadamard transform acts as a mathematical lens that rotates the data distribution, spreading the energy of massive outliers evenly across all channels and making the tensor safe for low-bit integer quantization.
- Outlier Unpacking: Rather than changing the distribution, Zhanpeng Zeng et al. proposed "unpacking" rare outlier values. Their framework, IM-Unpack, identifies values that exceed the integer capacity and mathematically decomposes them into multiple smaller integer entries. This allows the hardware to maintain high accuracy using standard 5-bit or 8-bit integer units without clipping critical information.
- Smart Grouping: Wenjin Guo et al. designed a hardware-friendly "ShiftQuant" quantizer that dynamically groups data to minimize precision loss, enabling sub-8-bit training from scratch.
Mechanism 3: Eliminating Floating-Point Bottlenecks
A truly integer-only training pipeline requires that every operation happens in integer math. Even if matrix multiplications are performed using integers, standard neural network architectures often contain hidden floating-point dependencies in non-linear layers, such as Batch Normalization, Layer Normalization, or Softmax. These layers require reductions, divisions, and exponential functions that typically force a temporary conversion back to floating-point.
To achieve full-stack integerization, these layers must be redesigned:
Standard Training Loop:
[Integer Matrix Mult] ──> [Floating-Point Batch Norm] ──> [Integer Backprop] (High Overhead)
Fully Quantized Training Loop:
[Integer Matrix Mult] ──> [Fixup Init / L1 Norm (Integer)] ──> [Integer Backprop] (Pure Integer)
For example, Xiaowei Chi et al. eliminated Batch Normalization entirely from the training loop. They replaced it with a specialized initialization strategy known as Fixup, allowing the entire forward and backward pass to run in 8-bit integer representations without losing stability. Alternatively, Wenjin Guo et al. substituted standard normalization with an L1-norm layer, which relies on simple absolute additions and bit-shifts rather than floating-point square roots.
For extreme cases like Binary Neural Networks, Luca Colombo et al. developed Binary Error Propagation (BEP), an algorithm that replaces the continuous backpropagation chain rule with purely bitwise logic and discrete integer updates.
Why It Works: The Surprising Robustness of Gradient Descent
The success of integer-only training hinges on a fundamental realization: the high dynamic range of floating-point numbers is a luxury, not a mathematical necessity for neural network convergence.
Empirical studies by Alireza Ghaffari et al. have shown that 8-bit integer arithmetic achieves accuracy nearly identical to standard floating-point training across diverse architectures, including vision transformers. Furthermore, Xishan Zhang et al. demonstrated that over 99% of gradients can be successfully compressed to low-bit formats using layer-by-layer dynamic precision adjustments.
Stochastic gradient descent is inherently noisy. The quantization errors introduced by restricting math to 8-bit or even 5-bit integers act as a form of stochastic regularization, which the network can naturally tolerate and adapt to during training. As long as the relative magnitudes of the gradients are preserved via shared exponents or dynamic scaling, the model will find its way down the loss landscape.
Hardware Realization at the Edge
Transitioning to a pure integer pipeline unlocks massive practical advantages when deploying models to resource-constrained environments.
On microcontrollers, Mark Deutel et al. demonstrated that executing both forward and backward passes in 8-bit integer format on Cortex-M cores slashes memory consumption by 34.8% and latency by 49% compared to simulated quantization methods. This eliminates the need for expensive runtime format conversions.
In distributed environments, Li Ding et al. applied full-stack low-bit integerization to federated learning, quantizing weights, activations, and gradients to sub-INT8 precision. This reduced communication bandwidth across edge devices by 8x while maintaining competitive accuracy.
For custom hardware, fixed-point architectures drastically simplify silicon design. Hardware accelerators built on FPGAs—such as those designed by Shreyas Kolala Venkataramanaiah et al. and Corey Lammie et al.—leverage 16-bit fixed-point arithmetic to execute full-pipeline training. By replacing power-hungry floating-point multipliers with compact integer logic, these systems achieve massive throughput gains and energy savings. As shown by the authors of an automatic compiler-based FPGA framework, shifting to fixed-point operations allows edge devices to achieve superior energy efficiency over traditional GPUs for small-batch on-device learning.
By reframing training as a discrete, integer-first process rather than a continuous floating-point calculation, this paradigm bridges the gap between resource-heavy AI research and the strict physical constraints of edge hardware.
Key Foundational Work
SPERT-II: A Vector Microprocessor System and its Application to Large Problems in Backpropagation Training (1995) — contributed 1 concept elements to this direction.
Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip (1990) — contributed 1 concept elements to this direction.