Modeling Quantization Error as Stochastic Noise

This direction is split into other directions: quantization-noise equivalence. Its last version stays readable and citable here.

Version 1 of 115 papers29 conceptsfrozen 2026-10-04

In traditional deep learning, quantization is viewed as a necessary evil—a source of "rounding error" that degrades model performance in exchange for memory and speed. We typically try to minimize this error by finding better scaling factors or using fine-tuning.

However, a growing body of research suggests a more powerful perspective: quantization error is not just error; it is a form of stochastic noise. By treating quantization as a controlled noise injection process, researchers have found they can mathematically "absorb" these errors into generative models, prevent divergence in Bayesian samplers, and even use pre-trained diffusion models to "denoise" compressed images back to high fidelity.

The Intuition: Quantization as a Forward Process

When we quantize a high-precision value to a discrete grid, we are essentially adding a signal-dependent noise term. In many iterative algorithms—like Langevin dynamics or Diffusion models—this looks remarkably similar to the stochastic noise already present in the system.

As Ruqi Zhang, Andrew Gordon Wilson, and Christopher De Sa demonstrate in Low-Precision Stochastic Gradient Langevin Dynamics, the failure of low-precision sampling isn't usually caused by bias (the "mean" error), but by overdispersion. The rounding process adds extra variance that makes the sampler explore a distribution that is "wider" than the target.

If we can model the exact distribution of this noise, we can compensate for it. This is the core idea behind variance-corrected quantization: instead of simple rounding, we sample from the discrete grid in a way that ensures the resulting noise matches a specific target variance or temperature.

Absorbing Noise into the Diffusion Schedule

Diffusion models are particularly well-suited for this perspective because they are built entirely on the management of Gaussian noise. In a standard diffusion process, we follow a "variance schedule" that dictates how much noise to add at each step.

When we quantize a diffusion model (Post-Training Quantization), the low-bit weights and activations introduce additional noise. In PTQD: Accurate Post-Training Quantization for Diffusion Models, Yefei He and Luping Liu propose that we can simply subtract this quantization-induced variance from the planned diffusion noise.

Essentially, if the quantization process is going to "accidentally" add noise, we should "deliberately" add less. By adjusting the schedule so the total noise (planned + quantization) matches the original training distribution, the model remains accurate even at very low bit-widths. Qian Zeng and Jie Song extend this in D2-DPM by modeling quantization noise as a time-step-aware Gaussian distribution, allowing it to be mathematically subtracted during inference.

Compression as a Generative Task

This "noise-centric" view of quantization also transforms how we think about image compression. Traditionally, lossy compression discards information, leaving behind "artifacts." But if we view those artifacts as a specific type of noise, we can use generative models to reverse the process.

In Lossy Image Compression with Foundation Diffusion Models, Lucas Relic and Markus Gross treat the quantization error of a codec as a starting point for a diffusion denoising process. Because the quantization error behaves like Gaussian noise, a pre-trained "foundation" model can treat the compressed image as a partially-noised sample and "denoise" it to recover high-frequency textures that were never actually stored in the bitstream.

Similarly, the OSCAR framework by Jinpei Guo and colleagues maps different compression bit-rates directly to "pseudo diffusion timesteps." A lower bit-rate simply corresponds to a state with more quantization noise, which the model then clears away in a single generative step.

Why It Works: The Statistical Buffer

The reason this direction is so effective—particularly in Bayesian sampling and generative modeling—is that these fields already rely on stochasticity.

  1. Momentum as a Buffer: As shown by Qifan Song and Ruqi Zhang in Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo, algorithms with momentum (like SGHMC) are naturally more robust to quantization noise than first-order methods. The momentum term acts as a low-pass filter, smoothing out the high-frequency "jitter" introduced by rounding.
  2. Exact Distributions: In Federated Learning, Mahmoud Hegazy and Aymeric Dieuleveut argue in Compression with Exact Error Distribution for Federated Learning that the shape of the error distribution matters more than its magnitude. If we can force quantization noise to be exactly Gaussian or Laplace, we can maintain the mathematical guarantees of privacy and convergence that would otherwise be destroyed by arbitrary rounding bias.

By moving away from the goal of "perfect reconstruction" and toward "statistical consistency," this research direction enables us to run our most complex models on hardware with significantly lower precision than previously thought possible.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_quantization_error_as_stochasti_f166ce41,
  title = {Modeling Quantization Error as Stochastic Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-quantization-error-as-stochastic-noise/rdr_f166ce4181fd449299ae74a14f0114e0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_quantization_error_as_stochasti_f166ce41_v1,
  title = {Modeling Quantization Error as Stochastic Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-quantization-error-as-stochastic-noise/rdr_f166ce4181fd449299ae74a14f0114e0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}