To run modern AI on edge devices like smartphones or smartwatches, researchers use quantization—the process of replacing high-precision 32-bit numbers with ultra-low precision versions, such as 1-bit (binary) or 2-bit (ternary) values. While this offers massive compression and speedups, it introduces a fundamental mathematical crisis: the functions used to "round" these numbers are piecewise constant. Their derivatives are zero almost everywhere, making standard backpropagation impossible.
The research direction across these papers focuses on overcoming the gradient mismatch—the discrepancy between the discrete operations used in the forward pass and the continuous approximations used to estimate gradients in the backward pass.
The Failure of the Straight-Through Estimator
The most common "hack" for training quantized networks is the Straight-Through Estimator (STE). During the forward pass, the model rounds a value to the nearest integer; during the backward pass, the STE simply ignores the rounding and treats the function as an identity mapping.
As noted by Chengxi Ye and Andrew G. Howard in Robust Training of Neural Networks at Arbitrary Precision and Sparsity, this creates a "lack of error awareness." The gradient calculation is oblivious to the quantization noise introduced in the forward pass. This mismatch leads to unstable training, slow convergence, and a significant accuracy gap compared to full-precision models. Jiancheng Lyu and Stanley J. Osher describe this in Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets as trying to navigate a loss landscape made of flat LEGO bricks—the "slope" provides no information for descent.
Softening the Step with Continuous Approximations
One primary strategy to bridge this gap is replacing the "jagged" discrete functions with smooth, differentiable surrogates that gradually sharpen during training.
In Quantization Networks, Xu Shen proposes using temperature-annealed Sigmoid functions to simulate quantization. Similarly, Luming Liang and colleagues introduced StableQAT StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths, which uses a "Rotated Damped Fourier Surrogate" (RDFS). By using smooth trigonometric functions instead of identity mappings, they provide a bounded, stable gradient that prevents the "exploding variance" often seen in ultra-low bitwidth training.
Statistical and Geometric Reframing
Rather than just smoothing the function, some researchers re-examine the statistical distribution of the weights and activations.
- Probabilistic Modeling: Oran Shayer and Ethan Fetaya developed LR-nets Learning Discrete Weights Using the Local Reparameterization Trick, which use the Local Reparameterization Trick to transform discrete weights into a smooth Gaussian approximation. This allows the model to learn the parameters of the distribution rather than the discrete values themselves.
- Geometric Manifolds: In Learning Discretized Neural Networks under Ricci Flow, Jun Chen treats the gradient mismatch as a geometric perturbation on a manifold. By applying Ricci Flow, the same mathematical tool used to solve the Poincaré Conjecture, the method "smooths out" the errors in the loss landscape over time.
Auxiliary Highways and Denoising
Another powerful intuition is to provide the quantized model with a "guide" during training—a full-precision path that is discarded at inference time.
Bohan Zhuang and Ian Reid proposed the Auxiliary Gradient Module [/paper/art_062d52f18f7c41e0812c19e3364fe597], which shares weights with the quantized network to provide a "gradient highway." This ensures that even if the quantized path produces a noisy signal, the auxiliary path keeps the weights moving in the right direction.
More recently, researchers at Google DeepMind have reframed quantization as a denoising problem Robust Training of Neural Networks at Arbitrary Precision and Sparsity. By modeling the rounding operation as additive noise and deriving gradients from a ridge regression objective, they create a "denoising dequantization transform." This approach treats the error not as something to be ignored (like STE) or smoothed (like Sigmoids), but as a signal to be explicitly modeled and corrected.
Why It Works: Information Retention
The shift in the field is moving from "minimizing quantization error" to "maximizing information retention." As Haotong Qin argues in DIR-Net, the goal isn't just to make the quantized numbers look like the original numbers, but to ensure the features they represent remain discriminative. By aligning the distributions of continuous features and discrete codebooks—using tools like the Wasserstein distance in Wasserstein VQ—researchers can ensure that the gradient signal remains meaningful even when the underlying data is reduced to a single bit.