Quantized neural networks are almost always designed and tested with "fake quantization": values are rounded to a low-bit grid, but every multiply and add still runs in exact 32-bit floating point. Homomorphic encryption lets a server run a network on data it cannot read. It does this with arithmetic that is itself noisy, because every ciphertext carries a small random error that grows with each operation. When a quantized network is run under the CKKS scheme, which packs thousands of real numbers into one ciphertext and computes on them approximately, values drift off the integer grid that fake quantization assumed. Keeping them on the grid requires evaluating a costly polynomial "rounding" step. We propose a fake-quantization simulator that also injects calibrated encryption noise and polynomial-rounding error. We then use it to answer one practical question: how much of a quantized network's tolerance for error can be spent on cheaper encryption parameters and fewer cleaning steps before accuracy collapses?
The opportunity is concrete. A quantization step of width $\Delta$ means the network already tolerates perturbations on the order of $\Delta$, and most encrypted-inference cost comes from protecting precision that this tolerance makes unnecessary. That cost includes large scaling factors, deep modulus chains, bootstrapping (the expensive procedure that refreshes a noisy ciphertext), and high-degree rounding polynomials. Current practice cannot exploit the tolerance safely. Clean fake quantization says nothing about encrypted accuracy, and running true encrypted inference on a validation set is far too slow to use inside a design or training loop. SHE reports about 2,258 seconds for a single CIFAR-10 image. We aim for a simulator that predicts encrypted accuracy to within half a point at plaintext speed, together with the cost savings it makes possible.
Core Claim
In CKKS inference of quantized networks, the error at the input to layer $l$ can be decomposed as
$$ e_l = e^{q}_l + e^{\text{poly}}_l(d_l) + e^{\text{ckks}}_l(s_l), $$
where $e^{q}_l$ is ordinary rounding error on a grid of step $\Delta_l$. The term $e^{\text{poly}}_l(d_l)$ is the error left by a degree-$d_l$ polynomial approximation of the rounding or step function, applied only if layer $l$ is "cleaned." The term $e^{\text{ckks}}_l(s_l)$ is the encryption and rescaling noise, which depends on the number of scaling bits $s_l$. Clean fake quantization models only the first term.
We claim three things, each testable. First, the second and third terms are non-negligible exactly in the regime where encrypted inference becomes cheap: small $s_l$, low $d_l$, and few cleaning points. In that regime, clean fake quantization overestimates encrypted accuracy. Second, a simulator that samples the last two terms from distributions calibrated on real ciphertexts predicts measured encrypted accuracy closely, and it is much cheaper than true encrypted evaluation. Third, searching over cleaning placement, polynomial degree and scaling bits with this simulator, and fine-tuning with its noise in the loop, cuts levels consumed and bootstraps at matched accuracy relative to cleaning every layer.
The design principle behind the third claim is that a network only needs to be snapped back onto its integer grid where accumulated drift approaches a fraction of $\Delta_l$. Everywhere else, drift is just more quantization-like noise that training can absorb.
Relation to Prior Work
Three existing works sit closest to this idea, and each assumes away the interaction we study.
HEQuant co-designs low-bit quantization with homomorphic encryption in two-party protocols. It packs several low-bit activations per coefficient, profiles the minimum plaintext modulus per layer, and identifies a "saturation barrier" below 4 bits. Its arithmetic is exact modular integer arithmetic, so quantized values never leave their grid. We target non-interactive CKKS, where drift off the grid is the central problem.
SHE avoids approximation entirely. It uses power-of-two weights and exact TFHE Boolean gates, which process one encrypted bit-level operation at a time without packing. It pays for that exactness in latency.
BLEACH supplies our core cleaning mechanism. It uses a polynomial step function to snap noisy CKKS values back to discrete states, which keeps Conway's Game of Life stable for over 50 iterations, whereas uncleaned CKKS fails after 4. BLEACH cleans after every step and was never applied to neural networks. Our question is where cleaning can be omitted or weakened because the network tolerates the drift.
Two further lines supply methods we reuse. Accumulator-Aware Post-Training Quantization bounds dot-product ranges with an $\ell_1$ constraint on weights so that narrow integer accumulators never overflow. We reuse that constraint for a different purpose. The rounding polynomial is accurate only on a bounded input interval, and inputs outside it are corrupted catastrophically, the CKKS analogue of overflow. An $\ell_1$ bound on weights therefore fixes the interval, and with it the polynomial degree needed. The polynomial-architecture work, Powerformer, PowerSoftmax and PolyNeXt, removes non-polynomial operations but evaluates its models in floating point. It serves as our unquantized encrypted reference.
The motivation is the general simulation-gap finding from low-precision training. Stable and low-precision training for large-scale vision-language models shows that simulated low precision hides instabilities that appear on native hardware, and both federated 8-bit studies (Low Precision Local Training is Enough for Federated Learning, Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point) caution that their gains are simulated. Encryption is a setting where the gap can be measured precisely, because a real encrypted run is the ground truth.
Research Program
The first component is calibration. For each primitive that appears in a quantized layer, we run real CKKS evaluations over many inputs and record the empirical distribution of decrypted error as a function of scaling bits, level and ring size. The primitives are the plaintext–ciphertext multiply, rotation-and-sum, rescale, bootstrap, and the step polynomial at each candidate degree. We fit simple parametric models to these distributions, Gaussian or heavy-tailed, and check how errors correlate across the packed slots of one ciphertext. The simulator then runs the quantized network in plaintext on a GPU. At each operation it adds errors sampled from these models, and at cleaning points it applies the exact polynomial, including its behavior outside the interval where it is accurate.
The second component is a search over encryption plans. A plan specifies, for each layer, whether to clean, which polynomial degree to use and how many scaling bits to allocate. Its cost is measured as levels consumed, number of bootstraps and wall-clock time on real hardware, not operation counts. We fit latency from measured primitive timings. The per-primitive TPU figures in Adapting AlphaEvolve to Optimize Fully Homomorphic Encryption on TPUs show why this matters: CKKS multiply and rotation have different costs after hardware tuning, 5,359 µs and 3,714 µs respectively on TPUv5e.
The third component is noise-aware fine-tuning. For a chosen plan, we fine-tune the quantized network with the simulator's noise in the forward pass, so the network learns to tolerate the drift the plan allows. Every selected plan is then validated by true encrypted inference on a held-out subset.
Baselines
The naive reference is clean fake quantization used as a predictor. We quantize a model with standard quantization-aware training, report its plaintext accuracy as the expected encrypted accuracy, and compare that prediction with the measured accuracy of the same model under each encryption plan. It is the default in the literature, so it is the right do-nothing point. Our simulator beats it if its prediction error is smaller by a clear margin in the cheap-parameter regime. We set that target at within 0.5 points of measured accuracy where clean fake quantization misses by more than 2 points. If clean fake quantization is already within 0.5 points everywhere, the first claim is false, and we would report that.
The strongest correctness-preserving competitor is BLEACH-style cleaning at every layer: an exact quantized network in CKKS with a step-function cleanup after each layer and a polynomial degree chosen conservatively, following BLEACH. It is strongest because it matches plaintext quantized accuracy by construction. The file reports no neural-network result for it (no reported result found on Lacuna for BLEACH on quantized CNNs), so we implement it ourselves on the same models, hardware and CKKS library. We give its polynomial degree the same search budget as our plans, restricted to the always-clean setting. Beating it means at least 30% fewer bootstraps or 25% lower wall-clock latency at no more than 0.5 points accuracy loss.
The strongest unquantized competitor is a floating-point polynomial network in CKKS, in the style of Powerformer. On GLUE tasks it reports 5.74 minutes per BERT-base inference on an A100 with no accuracy loss versus the unencrypted teacher, and a 70% reduction in bootstrapping relative to THOR. We compare on BERT-base with RTE, MRPC and SST-2 on the same A100 class of hardware, and we must beat 5.74 minutes at matched accuracy. For image models, the exact-logic reference is SHE, with 94.62% on CIFAR-10 at 2,258 seconds per image. We cite it for context only, since different hardware makes a latency comparison indicative rather than controlled.
Two ablations isolate the key ingredients. The first replaces calibrated encryption and polynomial noise with independent Gaussian noise of matched per-layer variance. The second keeps the calibrated noise but drops the $\ell_1$ range constraint, which shows whether the constraint is what makes low-degree cleaning safe.
Why not just…?
Why not just run real encrypted inference to evaluate each design? Real runs remain our ground truth for every final number. However, they cost minutes per sample even for well-engineered systems (5.74 minutes per BERT-base input in Powerformer). A search over hundreds of plans, or noise-in-the-loop fine-tuning, cannot be run that way. We will report the simulator's speedup over true evaluation alongside its prediction error, and the proposal is only worthwhile if both are good.
Why not just use analytic worst-case noise bounds to pick parameters? Worst-case bounds are safe, but they assume errors add up adversarially. Rounding a quantized value tolerates error up to about $\Delta/2$ per value, and averaged over a layer the errors largely cancel, so worst-case bounds over-provision scaling bits and cleaning. We will compare the plan chosen by worst-case bounds against the simulator's plan at equal measured accuracy. If the bounds already give plans within 10% of ours in cost, the simulator adds little.
Why not just add Gaussian noise during quantization-aware training? This is the real threat, and it is exactly our first ablation. If independent Gaussian noise with per-layer variance taken from calibration predicts and trains as well as the full simulator, the contribution shrinks to the calibration table plus the cleaning-placement search, which is still useful. We expect it to fail specifically at cleaning points. Low-degree step polynomials make large, structured errors near the edges of the grid and outside the interval where they are accurate, and a Gaussian of matched variance cannot represent those errors. The experiment will settle this.
Why not just use exact TFHE logic, as in SHE? Exactness removes the problem but gives up the packing of many values into one ciphertext. BLEACH measures about 70 µs per gate amortized in CKKS versus 13,420 µs in TFHE. The 2,258-second CIFAR-10 latency of SHE reflects that cost.
Proposed Experiments
We use ResNet-18 on CIFAR-10 and CIFAR-100 at 8-bit and 4-bit weights and activations, with activations replaced by low-degree polynomials so that each network runs entirely in CKKS. For a transformer, we use a Powerformer-style BERT-base on RTE, MRPC and SST-2 at 8 bits. A small PolyNeXt variant on a subset of ImageNet tests whether networks built entirely from polynomials, whose activations can explode per PolyNeXt, are more sensitive to encryption drift.
The metrics are the gap between predicted and measured accuracy, the rate of decryption or cleaning failures (values outside the polynomial's accurate interval), levels consumed, bootstrap count, and measured wall-clock time on A100 GPUs. On TPU we additionally record per-primitive timings. True encrypted evaluation uses 500 test inputs per plan, enough for accuracy confidence intervals of roughly ±2 points. The simulator uses the full test set.
The first decisive check is a map of prediction error across the grid of scaling bits, degree and cleaning density. It should show clean fake quantization diverging from measured accuracy as parameters get cheaper, while our simulator tracks it. The second is the Pareto frontier of accuracy against bootstraps and latency for simulator-searched, noise-fine-tuned plans versus clean-every-layer, which must dominate by the margins stated above. The third is the Gaussian ablation at cleaning points, which determines whether structured noise modeling is necessary. A negative result on any of the three is informative. It would show either that encryption drift is harmless in quantized networks, or that simple noise injection suffices.
References
- SHE: A Fast and Accurate Deep Neural Network for Encrypted Data
- HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference
- BLEACH: Cleaning Errors in Discrete Computations Over CKKS
- Accumulator-Aware Post-Training Quantization for Large Language Models
- Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption
- PowerSoftmax: Towards Secure LLM Inference Over Encrypted Data
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- Stable and low-precision training for large-scale vision-language models
- Low Precision Local Training is Enough for Federated Learning
- Towards Federated Learning with on-device Training and Communication in 8-bit Floating Point
- Adapting AlphaEvolve to Optimize Fully Homomorphic Encryption on TPUs