Noise-Proportional Quantization Error for Low-Bit Speech Enhancement

A speech enhancement model, a network that removes background noise from a recording, has a failure that language-model compression never faces. When its weights are rounded to a few bits, the rounding error lands directly in the audio. That error is largest exactly when the input was already clean. Towards Fully Quantized Neural Networks For Speech Enhancement observed this "clean signal paradox" at 8 bits. In their results, vanilla 8-bit Conv-TasNet lost about 2 dB in high signal-to-noise conditions, and they patched it with training on extra clean examples plus an auxiliary error-correcting block. Below 4 bits the rounding noise is far larger. Every working sub-4-bit speech result, such as 2-bit Conformer quantization for automatic speech recognition, is for recognition, where the output is a discrete word choice that absorbs numerical noise. Nobody has shown low-bit enhancement that stays transparent on clean speech.

This note proposes a structural fix rather than another rounding scheme. We change what the quantized network is asked to output. It no longer predicts the whole clean spectrum or a full multiplicative mask. Instead it predicts only a gated correction to the untouched noisy input, and the input bypasses the low-bit path at full precision. Under this design, quantization error scales with the amount of noise being removed rather than with the loudness of the speech. Clean input passes through nearly unchanged even at 2–4 bit weights. In two sentences: low-bit enhancement fails on clean audio because the rounding error rides on the speech itself. If the low-precision network only ever outputs a correction to a full-precision copy of the input, its error shrinks as the input gets cleaner.

Core Claim

The claim rests on a simple error model. Consider a standard mask-based enhancer that outputs $\hat S = M \cdot X$, where $X$ is the noisy spectrum, $S$ the clean speech, and $M$ the predicted mask. Quantizing the network perturbs the mask by a relative error $\delta$, which grows sharply as the bit-width drops. At high input SNR the ideal mask approaches one and $X \approx S$. The output error $\delta \cdot X$ then has energy proportional to the speech, so the output SNR saturates near $-20\log_{10}|\delta|$ however clean the input was. This is the clean signal paradox written as a ceiling.

We propose instead

$$\hat S = X + g(X)\,\Delta(X),$$

where $\Delta$ is the low-bit network's predicted correction, $g \in [0,1]$ is a learned gate on each time-frequency bin, and $X$ is carried on a full-precision skip path that is never quantized. The ideal correction has magnitude on the order of the noise $N$. Quantization error therefore enters as roughly $g\,\delta\,|N|$. The resulting output SNR is approximately

$$\mathrm{SNR}_{\text{out}} \approx \mathrm{SNR}_{\text{in}} - 20\log_{10}(g\,|\delta|),$$

which keeps rising with input SNR instead of hitting a ceiling. The relation follows directly from the error entering in proportion to $|N|$ rather than $|S|$.

The same argument applies to phase. MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra predicts the clean phase outright. At high SNR the noisy phase is already nearly correct, and a quantized phase decoder would replace it with a rounded estimate. We instead predict a gated phase offset added to the noisy phase. The concrete, testable claim has two parts. First, this reparameterization flattens the curve of quality loss versus input SNR under 4-bit and 3-bit weight quantization. Second, it moves the bit-width at which enhancement collapses lower than the current recipes do, at almost no extra compute.

We also expect the reparameterization to give little or no benefit in full precision. If it does, that supports the view that the gain is specific to quantization and is not just a better architecture. If it fails to hold, that weakens the error-model explanation.

Research Program

The work has three parts. The first is diagnostic. We take MP-SENet, a 2.05M-parameter model with parallel magnitude and phase decoders, and Conv-TasNet as used in the LibriMix and FQSE studies. We quantize each at 8, 4, 3 and 2 bits for weights and 8 and 4 bits for activations. We then measure degradation separately within input-SNR bins from 0 dB to 40 dB, plus perfectly clean input. We also quantize the magnitude decoder, phase decoder and shared encoder one at a time to see which part reaches the accuracy cliff first. This directly tests the open question of whether phase, a periodic and discontinuous quantity, breaks before magnitude.

The second part is the method. We rewrite both decoders in gated-residual form. The magnitude decoder outputs a bounded log-gain correction, and the phase decoder outputs an angular offset, both multiplied by the gate. The skip connection and the final addition run at the input's native 16-bit precision. Even on an all-integer chip this adds only one elementwise operation per output bin. It also sidesteps the input/output precision bottleneck that FQSE had to solve with its input splitter, though we still need the splitter on the network's input branch. Training uses quantization-aware training, meaning the rounding is simulated during training so the network learns around it. We use the learned step-size method that FQSE uses. We add a gate penalty that pushes $g$ toward zero on bins where the full-precision teacher's correction is small.

The third part is composition. A rewritten output does not remove outliers inside the network, the rare very large weights that make low-bit rounding hard. We therefore stack our parameterization on each of three competing outlier treatments. The first is greedy clipping as in the 2-bit Conformer work. The second is random rotations as in QuIP: 2-Bit Quantization of Large Language Models With Guarantees. The third is keeping the most important parameters in full precision as in Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models. This also gives the first comparison of these three treatments on a model that reconstructs a signal rather than choosing tokens.

One prior result constrains how much we should expect. Rybakov et al. found that models under roughly 100M parameters collapse at 2 bits, and MP-SENet is about 50 times smaller. We therefore treat 4-bit and 3-bit operation as the main target and 2-bit as a stretch goal. We will report a 2-bit collapse honestly if it happens.

Baselines

The naive reference point is plain quantization-aware training of the unmodified MP-SENet and Conv-TasNet at each bit-width, using the same learned step-size quantizer and per-channel scales. This shows the size of the cliff we are trying to move. No reported result found on Lacuna for MP-SENet at any reduced precision. The full-precision anchors are PESQ 3.50 on VoiceBank+DEMAND (MP-SENet, MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra) and wide-band PESQ 3.60 for the extended version in Explicit estimation of magnitude and phase spectra in parallel for high-quality speech enhancement. We will retrain these ourselves rather than trust cross-paper numbers.

The strongest existing competitor is the FQSE recipe of high-SNR oversampling plus the residual quantization block, which we extend to 4, 3 and 2 bits. We combine it with the Rybakov et al. low-bit techniques of asymmetric levels, per-sub-channel scales and greedy clipping. FQSE is the only method aimed at the clean signal paradox, and Rybakov's recipe is the only proven sub-4-bit recipe for speech transformers, so together they are the hardest version of "existing tools suffice." FQSE's reported numbers are SI-SNR 14.77 dB for its 8-bit Conv-TasNet on LibriMix, against 14.74 dB for full precision and 14.42 dB for vanilla 8-bit (Towards Fully Quantized Neural Networks For Speech Enhancement).

We first reproduce Conv-TasNet at 8 bits on LibriMix train-360 and need to land within a few tenths of a dB of those figures before trusting the extensions. All methods get the same data, the same number of training steps and the same budget of learning-rate and clipping trials. FQSE's extra block costs about 15% more compute, so we either give our model matching extra width or report the comparison at matched operation counts.

Two results would count as beating this baseline. The first is clearly lower degradation from full precision in the 30 dB-and-above and clean-input bins at 4 and 3 bits. The second is no loss at low SNR, on matched memory and compute. A gain that appears only on average over all SNRs, and not in the clean bins, would not support the error model.

Two ablations isolate the key ingredient. The first keeps the residual form but removes the gate. The second keeps everything but routes the skip path through the quantized pipeline, which should bring the saturation back if the full-precision bypass is what matters. A third control applies the residual form to the full-precision model, as explained in the Core Claim.

Why not just…?

Why not just stay at 8 or 4 bits, which already work? FQSE does show 8-bit parity on Conv-TasNet. Language-model studies such as The case for 4-bit precision: k-bit Inference Scaling Laws find 4 bits optimal for a fixed bit budget. That finding is itself an argument for running a larger model at low precision instead of a small one at high precision, but it was shown for token prediction, not signal reconstruction. Our diagnostic sweep reports where the cliff actually sits for enhancement. If 4-bit MP-SENet already shows no clean-input penalty under vanilla training, the motivation shrinks to the 3-bit and 2-bit regimes, and we will say so.

Why not just keep the output heads in high precision (mixed precision)? This is a real threat. CherryQ and CRVQ: Channel-relaxed Vector Quantization for Extreme Compression of LLMs both show that protecting a small set of sensitive parameters recovers much of the loss. Our error model predicts that a high-precision head alone cannot fix saturation. The head's input features are already corrupted by the quantized trunk, so the mask error stays proportional to the speech. We test this directly by running an 8-bit-heads baseline at matched total memory. If it closes the clean-input gap, our contribution reduces to a diagnosis.

Why not just blend the enhanced output with the noisy input after the fact? A post-hoc dry/wet mix, $\alpha \hat S + (1-\alpha) X$ with $\alpha$ set from an SNR estimate, is the cheapest imitation of our gate. It is also a serious alternative. Its weakness is that blending reduces the quantization error and the noise suppression together and in the same proportion. Our gate is trained under quantization to suppress only where correction is needed, and it can differ for each time-frequency bin. We will tune $\alpha$ on the test SNR bins, as an upper bound on the blend, and compare the resulting quality-versus-SNR curves.

Why not just use a smaller full-precision model at the same memory? Compact models such as PCNN: A Lightweight Parallel Conformer Neural Network for Efficient Monaural Speech Enhancement are the natural edge alternative. We train an 8-bit PCNN-sized model at the same weight memory as our 3-bit MP-SENet and compare the two. Losing this comparison would mean low-bit quantization of enhancement is not worth doing at this model scale.

Proposed Experiments

The main benchmarks are VoiceBank+DEMAND, scored with PESQ and the composite quality scores CSIG, CBAK and COVL, and LibriMix, scored with SI-SNR improvement. LibriMix allows a direct check against FQSE's reported Conv-TasNet numbers. The synthetic test set of the INTERSPEECH 2020 Deep Noise Suppression Challenge serves as a held-out generalization check, and a small P.808 crowdsourced listening test, the challenge's subjective rating protocol, covers the 4-bit and 3-bit systems.

Beyond the standard splits, we remix DNS clean speech with DEMAND noise at controlled SNRs from 0 dB to 40 dB, plus a clean-input set. For each method and bit-width we report two things. One is the slope of degradation against input SNR. The other is a "transparency" score: output SNR against the reference when the input is clean.

We also replay the sparse-overlap condition of LibriMix: An Open-Source Dataset for Generalizable Speech Separation. There, separation models lost to a simple identity mask, the setting in which a learned near-identity output should help most.

The decisive check is the shape of the curve. Under vanilla and FQSE-style training at 4 and 3 bits, output SNR should flatten at high input SNR. Under the gated residual, it should keep rising, and turning off the full-precision bypass should restore the flattening. If instead both curves flatten together, the error model is wrong and the proposal fails cleanly.

Secondary checks cover two further questions. One is whether the phase decoder breaks before the magnitude decoder, measured with phase-distance metrics alongside PESQ. The other is which outlier treatment (clipping, rotation or protected parameters) works best inside a model that reconstructs a signal.