Decorrelating Quantization Error in Neural Simulators Used as Synthetic Data Engines

Neural surrogates are networks trained to imitate slow physics solvers. Fourier Neural Operators are one example: they learn the solution map of a partial differential equation in frequency space. Their main appeal is cheap synthetic data. A surrogate that runs 100–400× faster than a numerical solver can label millions of simulated scenes, and those labels can then train a second, downstream model, such as a seismic inversion network or a source locator. Quantization would make these generators cheaper still. Quantization means storing weights and computing activations with a few integer bits instead of 32-bit floats. The catch is that every sample the quantized generator produces carries its rounding error, and the downstream model learns that error as if it were physics.

Our claim is that what harms the downstream model is error that repeats in the same direction across the whole dataset, not the size of any single sample's error. A deterministically quantized surrogate makes the same input-dependent mistake every time, and the downstream model absorbs it as a bias. If the surrogate's quantization is randomized from sample to sample, the errors become roughly independent and zero-mean. Training on many samples then averages them out. We propose to randomize in two ways: subtractive dither on activations, and an ensemble of differently rotated low-bit weight copies. Under a fixed generation budget, this should let 4-bit and even 2-bit surrogates produce better downstream models than either float surrogates or exact solvers, because the cheaper generator buys many more samples without systematic contamination.

What exists and what is missing

Two papers quantize neural PDE surrogates, and both judge success by the surrogate's own error. Efficient Neural PDE-Solvers using Quantization Aware Training trains FNOs, U-Nets and Transformers with simulated rounding and an MSE loss. It shows 8-bit weights and activations cost "virtually no increase" in MSE, and that quantized high-resolution models beat downscaled float models at equal compute. Adaptive Mesh Quantization for Neural PDE Solvers assigns 4- or 8-bit precision per mesh node using a learned difficulty predictor. These papers motivate our work: they establish that quantized surrogates are useful. Neither asks what happens to a model trained on the data the surrogate produces, and neither goes below 4 bits.

On the data side, GenQ uses diffusion-generated images to calibrate quantized classifiers. Its data flows in the opposite direction to ours: synthetic data serves quantization, whereas here quantization serves synthetic data. We borrow its idea of filtering bad synthetic samples as a baseline.

Two Float32 physics corrections constrain our design. ECF enforces conservation by replacing the zero-frequency Fourier mode. It can only remove error in that one mode, so it cannot by itself remove a bias spread across all frequencies. Buckingham π projection shows surrogates break under unit and scale shifts. For that reason we hold the parameter ranges fixed and do not confound our study with scale shift.

From the quantization literature we reuse a technique. QuIP multiplies weights by random orthogonal matrices so that large outlier values are spread evenly across coordinates, and this makes 2-bit LLMs usable. We use the randomness of those rotations for a second purpose: rotation draws give different error patterns, so an ensemble of rotated copies has decorrelated errors. The known collapse below 3 bits (Evaluating Quantized LLMs, LLaMA3 quantization study) tells us that 2-bit surrogates will have large per-sample error. That is exactly the regime where the bias-versus-variance distinction should decide the outcome.

Core Claim

Let $G$ be the exact solution operator, $a_i$ a sampled input such as a velocity field or permeability map, and $\varepsilon_i$ the error the quantized surrogate adds to $G(a_i)$. A downstream model trained on pairs $(G(a_i)+\varepsilon_i, a_i)$ converges toward a target shifted by the conditional mean error:

$$ b(a) = \mathbb{E}[\varepsilon \mid a]. $$

Here $b(a)$ is the error the generator makes consistently at input $a$. With deterministic rounding, $\varepsilon$ is a fixed function of $a$, so $b(a)=\varepsilon(a)$ and the full error becomes bias. Subtractive dither changes this. Uniform noise is added before rounding and removed after, and classical signal-processing results say the resulting rounding error is then independent of the signal and has zero mean. Per-layer error therefore contributes variance, not bias, to the output, apart from second-order terms introduced by the network's nonlinearities. Training on $N$ samples suppresses that variance roughly as $1/N$.

The prediction follows. At matched generation cost, randomized low-bit generators should shift the best compute–accuracy trade-off for downstream models toward fewer bits. Deterministic low-bit generators should hit a floor, set by the size of $b$, that no amount of extra data removes.

Research Program

We build three things.

The first is a set of quantized FNO generators at 8, 4, 3 and 2 bits. We train them with the QAT recipe of van den Dool et al., using QuIP-style random rotations of the real and imaginary Fourier weight tensors for the sub-4-bit settings.

The second is two randomization schemes. The activation scheme applies subtractive dither, with a fresh random seed for each generated sample. The weight scheme keeps $K$ copies of the weights, each quantized under an independent rotation draw, and picks one copy at random per sample. Memory grows only by $K$ low-bit copies; for example, four 2-bit copies occupy the same memory as one 8-bit model.

The third is an analysis tool that splits surrogate error over the test distribution into a bias part and a variance part. We estimate the bias part by generating many randomized outputs for the same input. We then check whether the bias part predicts downstream degradation better than surrogate MSE or the PDEBench conservation and spectral errors (cRMSE, fRMSE) do.

Downstream tasks are inverse problems in which the surrogate supplies the training data:

  • permeability from pressure on PDEBench 2D Darcy flow (PDEBench);
  • viscosity and initial condition from snapshots on PDEBench 1D Burgers and 2D Navier–Stokes;
  • velocity model from wavefields in the seismic setting of Yang et al..

For the seismic task we regenerate the Gaussian-random-field velocity models and solver labels following their description. Every downstream model is tested only on exact-solver data, so the generator's error can only hurt and never matches the test labels.

Baselines

The naive reference is a downstream model trained on exact numerical-solver data under the same generation budget. Because the solver is slow, this means fewer samples. A float32 FNO generator under the same budget is the natural "no quantization" point. It is a strong competitor because Yang et al. report a full-waveform-inversion relative $L_2$ misfit of 0.0319 for FNO against 0.0289 for the spectral-element solver. That is evidence that float surrogate data is nearly solver-quality.

The strongest existing quantized method is deterministic QAT following van den Dool et al., given QuIP rotations below 4 bits so that it is not handicapped. We found no reported result on Lacuna for downstream models trained on data from any quantized surrogate, so the comparison numbers will come from our own runs.

Fairness rules apply to all arms. Every arm shares the same generator architecture, training data, QAT schedule, downstream architecture, and downstream tuning budget. Generation budget is matched under two cost measures, both always reported. The first is the cost proxy $M \cdot b_w \cdot b_a$ from van den Dool et al., where $M$ is the operation count and $b_w$, $b_a$ are the weight and activation bit-widths. The second is measured wall-clock time on hardware with native int8/int4 kernels. This matters because 2- and 3-bit formats lack native GPU support, and the proxy alone would flatter them.

We add two physics-correction baselines:

  • staged ECF applied to the quantized generator's outputs;
  • GenQ-style filtering that discards samples with high PDE residual or high conservation error.

The key ablation compares deterministic and randomized quantization at identical bit-width and identical sample count. This isolates decorrelation as the active ingredient. A further ablation runs dither without subtraction, which adds noise but does not make the error signal-independent. It separates genuine decorrelation from mere data augmentation.

The proposal wins under three conditions. First, randomized 4-bit or lower generators must beat both float32-FNO data and exact-solver data in downstream error at matched wall-clock budget. Second, their downstream error must keep falling as sample count grows. Third, the deterministic counterpart must plateau.

Why not just…?

Why not just use 8-bit, which van den Dool et al. found essentially lossless? At 8 bits the bias may well be negligible, and we expect randomization to give little benefit there. That outcome would be informative, not fatal. The cost proxy falls by 4× from w8a8 to w4a4 and by 16× from w8a8 to w2a2. Whether those extra samples can be used is precisely the open question, because the 4-bit and lower regime is where the existing evidence stops.

Why not just fine-tune the downstream model on a small set of exact-solver samples? This is a real threat. A few hundred exact samples might correct a consistent bias directly. The LLaMA3 study's finding that small-set fine-tuning can hurt (Huang et al.) comes from a different domain and does not settle the question here. We therefore run an "anchor" arm: deterministic low-bit data plus fine-tuning on 1–5% exact data, charged to the same budget. We also run randomized data plus the same anchor. If the anchor alone closes the gap, we will report that randomization adds nothing beyond it. The bias–variance analysis predicts the anchor helps most when the bias is low-dimensional and smooth, and least when it is high-frequency and input-dependent.

Why not just filter or correct the bad samples? Residual filtering removes outliers. It cannot remove a consistent bias, because every sample carries the bias and all of them pass the filter. ECF corrects only the zero-frequency mode (ECF). Our decomposition will show how much of the bias lies outside the zero-frequency mode, and therefore how much ECF is able to remove. The filtering and ECF baseline arms measure this directly.

Decisive Checks

The central test is the downstream error curve as the number of generated samples grows, plotted for each bit-width with deterministic and randomized generators side by side. The thesis holds if three things hold together:

  • deterministic curves plateau at a level that tracks the measured bias norm;
  • randomized curves keep descending;
  • the bias norm predicts downstream degradation better across settings than surrogate MSE, cRMSE or fRMSE do.

The thesis fails in either of two cases. One is that randomization shows no gain once the no-subtraction-dither and anchor arms are controlled for. The other is that 2-bit randomized generators are so noisy that the variance never averages out within realistic dataset sizes. If either happens, we will report the bit-width at which synthetic-data utility breaks. That threshold is itself a missing result for practitioners deploying quantized simulators.