Approximate Multipliers for Low-Bit Language Models, Budgeted by the Hardware Noise Floor

Chip designers can save a large share of arithmetic power by using approximate multipliers, circuits that skip the least significant partial products of a multiplication. On convolutional networks, a cheap analytical correction makes this nearly free without retraining the model. Nobody has done this for large language models. Nobody has said how much error is acceptable, either. The usual rule of "under 1% average accuracy drop" hides which individual answers change.

This proposal uses a yardstick that LLM practitioners already accept implicitly: moving the same model from one GPU to another, or changing the batch size, already flips some answers. We propose to (i) carry the control-variate correction of approximate multipliers into 4-bit integer LLM inference, aligned with the weight groups that quantization already uses, and (ii) choose per-layer approximation levels so that the remaining effect on outputs is statistically no worse than this cross-hardware noise floor.

Core Claim

In weight-quantized integer matrix multiplication with asymmetric zero-points, perforated multipliers produce a large, predictable bias. The stored integer weights are non-negative, with a group mean far from zero. That bias can be removed almost exactly by a per-group mean correction. The correction costs one extra sum per token per weight group, shared across all output channels.

With this correction, we claim a quantized 7B model can run its linear layers on perforated multipliers and stay within the cross-GPU answer-flip floor. The flip floor will be measured with paired exact McNemar tests, a significance test for paired right/wrong outcomes, following When LLMs get significantly worse. On the same matched-energy model, this setting should Pareto-dominate simply dropping to exact 3-bit or 2-bit weights.

The accumulator is integer, so the approximate system is also bit-reproducible. Its deviation from the reference model is fixed, not random. We treat that as a side property of integer accumulation, not as our contribution. It does, however, change how deviations should be compared: a fixed, repeatable bias against hardware jitter that varies from run to run.

Why this intersection, and what already exists

Three strands of prior work supply the method.

  • Control variate approximation. Control Variate Approximation for DNN Accelerators and Leaveraging Highly Approximated Multipliers in DNN Inference model the error of a perforated multiplier as the weight times the dropped activation bits. They cancel its expected value with an extra accumulator column that uses the filter's mean weight. They report 24% power savings at 0.16% loss and up to 45% savings under 1% loss, but only on 8/16-bit convolutional networks.
  • Error-cancelling pairs. Positive/Negative Approximate Multipliers pairs multipliers with opposite-sign errors so they cancel within a dot product. That pairing depends on a fixed weight-to-multiplier mapping.
  • Approximate multipliers plus low-bit quantization. FAMES is the closest prior work. It selects approximate multipliers per layer for mixed-precision quantized networks down to 2 bits. It uses a Taylor estimate of the loss and an integer linear program, and reports 28.67% energy reduction within 1% loss on CIFAR-10 and ImageNet ResNet-18. FAMES applies no error correction, is limited to CNNs, and uses an average-accuracy threshold.

Our proposal differs from all three in four ways. It adds a correction keyed to quantization groups. It targets Transformer linear layers, where the same activation row is reused by thousands of output channels. It is evaluated on long autoregressive generation. And its acceptance criterion is a measured noise floor rather than a fixed percentage.

Two papers motivate using the noise floor as the budget.

Together these imply a deployment tolerance exists. Nobody has used it as a design target for hardware savings. Total Variation Distance Estimation in Autoregressive Models supplies a second, distribution-level measure of divergence. It reports 0.586 total variation distance (the maximum difference in probability two distributions assign to any event) between vLLM and sglang over 500 tokens on Qwen3-0.6B.

Several papers constrain the design.

  • Quantization methods. QuIP rotates weights with random orthogonal matrices so that outliers spread evenly across dimensions. Its 2-bit Llama 2 70B reaches 6.33 WikiText-2 perplexity. CherryQ keeps the 0.39% most important weights in FP16 and reaches 9.55 C4 perplexity at 2 bits on Llama2-7B.
  • Why our correction depends on asymmetric integers. Rotation makes real-valued weights roughly zero-mean, so a mean-based correction would remove nothing there. The bias we correct exists only in the non-negative integer weights produced by asymmetric quantization.
  • Mixed-precision weights. CherryQ's FP16 "cherry" weights must bypass the approximate path entirely.
  • Where determinism comes from. Causes and Effects of Unanticipated Numerical Deviations shows that even FP64 does not remove cross-platform divergence, because reduction order varies. That supports keeping the approximated linear layers on exact integer accumulators. Attention softmax and normalization remain floating point, so the system as a whole is not deterministic. We will report both components separately.

The correction

Consider output channel $j$ of a linear layer with integer weights $W_{ij}$ in groups $g$ of 128 input channels. Suppose the multiplier perforates the $m$ lowest activation bits, with dropped part $x_i^{\text{low}}$. The error is $e_j=\sum_i W_{ij}x_i^{\text{low}}$.

We subtract the group-mean estimate, where $\mu_{gj}$ is the mean integer weight of group $g$ in channel $j$:

$$\hat e_j=\sum_g \mu_{gj}\sum_{i\in g}x_i^{\text{low}}.$$

This leaves the residual

$$r_j=\sum_g\sum_{i\in g}(W_{ij}-\mu_{gj})\,x_i^{\text{low}}.$$

The cost is low because the inner sums over $x_i^{\text{low}}$ are computed once per token and reused by every output channel. The residual is zero-mean, with variance that scales with the within-group spread of the weights. That spread is bounded by the narrow 4-bit range.

The group means $\mu_{gj}$ come directly from the stored weights. No activation profiling and no retraining are needed. This removes the calibration dependency that the CNN versions still carry.

Research Program

Hardware emulator. We will build a bit-accurate emulator of W4A8 and W4A16-integer linear layers with perforated multipliers at $m\in\{1,2,3,4\}$, with and without group correction. Per-operation energy will come from synthesized multiplier and accumulator designs, including the cost of the correction path.

Measuring the noise floor. For each model we will first measure the floor. We will run the BF16 reference on several GPU types and batch sizes and record three things: the paired flip rate per benchmark, the total variation distance per prefix length, and the token index at which greedy outputs first diverge.

Choosing perforation levels. We will then choose a perforation level per layer by a FAMES-style integer program. The constraint is that a cheap estimate of each layer's logit perturbation, summed across layers, must stay below the floor. We will validate end to end with McNemar tests against the floor.

Analysis. Finally, we will analyze where the approximation error enters long reasoning chains. We will compare its divergence profile with that of hardware jitter to see whether a fixed bias does more or less damage than random reordering at the same per-token magnitude.

Baselines

Unmodified quantized model. The "do nothing" reference is the same W4 quantized model, quantized with the same GPTQ-style or QuIP rounding, on exact integer multipliers. It sets the energy and accuracy anchor. Every other method shares its weights, calibration data, and evaluation prompts.

Uncorrected perforation. This is the same perforation schedule with no correction. It isolates the contribution of the group control variate. The CNN work reports correction gives a 1.9x improvement over uncorrected approximation (Leveraging Highly Approximated Multipliers); we test whether a comparable gap appears in LLMs.

Per-channel mean correction. This uses a single mean per output channel, as in the original CNN method, instead of per-group means. It is the ablation for our specific ingredient.

FAMES applied to Transformers. The strongest existing method is FAMES, run on the same layers with the same energy model and the same integer-program solver and budget. Its reported result is 28.67% energy reduction within 1% accuracy loss on CIFAR-10 and ImageNet ResNet-18. We found no reported result on Lacuna for approximate multipliers on any LLM.

Lower-bitwidth exact quantization. The quantization-side competitor is exact W3 and W2 quantization with QuIP and CherryQ. The reported reference points are QuIP's 6.33 WikiText-2 perplexity for 2-bit Llama 2 70B and CherryQ's 9.55 C4 perplexity for 2-bit Llama2-7B.

What counts as beating them. We beat FAMES if we deliver more energy savings at a statistically non-significant flip rate relative to the noise floor. We beat lower-bit quantization if our points lie strictly on the better side of the matched-energy Pareto curve.

Why not just…?

Why not just quantize to fewer bits on exact hardware? This is the most serious threat. Going from W4 to W3 shrinks every multiplier, and it may save as much energy as perforation does. We do not argue around this. The matched-energy Pareto comparison against QuIP and CherryQ at 3 and 2 bits is the central experiment. The sub-3-bit cliff that Evaluating Quantized Large Language Models and An empirical study of LLaMA3 quantization document for reasoning tasks suggests perforating W4 may degrade more gracefully. However, combining W3 with light perforation may be best. We will report that combined setting too.

Why not just reuse FAMES's selection without any correction? On CNNs, the correction alone accounts for large accuracy gains: up to 21% on average over uncorrected approximation in Control Variate Approximation. In LLMs, the bias from non-negative integer weights accumulates across 4096-wide dot products, so we expect the gap to widen. The no-correction baseline at matched energy settles this directly.

Why not just judge by a fixed 1% average-accuracy drop? Average accuracy can hold steady while many individual answers change. Kübler et al. show that even 1.3% to 2.8% flips can be benign or harmful depending on how they are paired. Calibrating to the measured hardware floor gives a tolerance that already passes in practice. We will report both criteria to show where they disagree.

Why not just upcast to FP32 for reproducibility, as LayerCast does? LayerCast restores determinism, but it requires full-precision multipliers, which is the opposite of saving power. Our integer path gets reproducibility of the linear layers for free. We will report LayerCast as a reproducibility reference, not as an energy competitor.

Proposed Experiments

Models and benchmarks. We use Llama2-7B and Llama2-13B for perplexity on WikiText-2 and C4, matching the CherryQ and CRVQ setups. We use DeepSeek-R1-Distill-Qwen-7B on AIME'24 and MATH500, matching the Yuan et al. setup, together with an Open LLM Leaderboard v2 subset (BBH, IFEval, MMLU-Pro) for paired McNemar tests.

Metrics. We report perplexity, paired flip rate with exact McNemar p-values, total variation distance against the BF16 reference as a function of length, first-divergence token index, and emulated energy per token for linear layers.

Decisive checks.

  1. Can group correction at $m=2$ or $m=3$ keep flips non-significant relative to the cross-GPU floor, while uncorrected perforation and FAMES at equal energy cannot?
  2. At matched energy, do corrected-W4 points dominate exact W3 and W2 in both perplexity and reasoning accuracy?
  3. In long chains of thought, does a fixed approximation bias cause earlier or later divergence than hardware jitter of equal per-token total variation distance?

If the first check fails, the noise-floor criterion is too strict for perforation in LLMs. If the second fails, the honest conclusion is that lower-bitwidth quantization dominates and approximate multipliers have no role in this regime.