Homomorphic encryption lets a server run a neural network on data it cannot read. It is also 3 to 5 orders of magnitude slower than ordinary inference. Much of that cost is a precision budget. The widely used CKKS scheme, a homomorphic scheme that does approximate arithmetic on real numbers, keeps every value at a chosen number of bits of precision (the "scale"). Each multiplication consumes one "level" of a large ciphertext modulus. When the levels run out, an expensive refresh called bootstrapping is needed. Compilers such as CHET, EVA and Orion pick these parameters conservatively, so that the encrypted network matches the plaintext network. They never ask the network to meet the cryptography halfway.
We propose to borrow the central trick of analog-hardware-aware training: during training, inject a calibrated model of the device's errors so that the weights land where those errors do not matter. We apply that trick to the rounding and rescaling error of CKKS. A network trained this way should keep its accuracy at much lower scale bits. Lower scale bits mean smaller moduli and more multiplications per bootstrap, which means fewer bootstraps and faster encrypted inference. In two sentences: treat CKKS precision loss as a noisy accelerator and train for it. Then let each block of the network run at the cheapest precision it tolerates, and recover the few inputs that fail by re-running them at full precision.
Core Claim
The claim is that, for polynomial-only networks, the accuracy-versus-scale-bits curve under CKKS is a property of training and can be moved. It is not fixed by the architecture. A network trained with block-wise, calibrated CKKS noise can run with substantially fewer scale bits per level than the same network trained normally, at equal encrypted accuracy. The saved bits translate directly into fewer bootstraps and lower latency in an existing compiler.
This is different from the closest work. A Methodology for Training Homomorphic Encryption Friendly Neural Networks trains quadratic activations $ax^2+bx$ with a smooth ReLU-to-polynomial transition and distillation. That makes the network expressible in encryption, but it trains in exact floating point and treats encryption as a fixed backend. Activation-Free Backbones for Image Recognition (PolyNeXt) builds strong activation-free backbones, reporting 83.9% ImageNet Top-1. However, it never runs on ciphertexts and never models encryption error. The compilers, CHET, EVA and Orion, optimize parameters, packing and bootstrap placement for a network that is already trained. None of these closes the loop from encryption error back into training. That missing loop is the proposal.
Research Program
The method comes from analog hardware-aware training. aihwkit keeps exact "shadow" weights, injects device non-idealities in the forward pass, and updates with exact gradients. We reuse this loop directly. Two follow-up results constrain how the noise should be designed. First, TRICE shows that zero-mean symmetric noise is suboptimal for worst-case behaviour; right-censored noise raised ResNet-18 1st-percentile accuracy by 26.01%. Second, BayesFT shows that layer-wise noise tuned by Bayesian optimization greatly outperforms uniform injection. Both results warn us against assuming that a generic Gaussian will do.
The first component is a measured CKKS error model. This plays the role that a device model plays in analog work. We will run each primitive (plaintext-ciphertext multiply, rotate-and-sum, rescale, polynomial evaluation, bootstrap) many times in a CKKS library. We will do this across a grid of scale bits and ring dimensions and record the error of the decrypted output relative to exact arithmetic. We then fit an error distribution per operation type, conditioned on the input magnitude. We do not assume a closed form, because bootstrapping error in particular is best measured rather than derived.
The second component is noise-matched training. Let $b$ index the network's blocks, and let $s_b$ be the scale bits used in block $b$. Let $\epsilon(s_b)$ be noise drawn from the fitted error model and applied at every simulated homomorphic operation in that block. Let $C(s)$ be the compiler's estimated latency for the precision assignment $s=(s_1,\dots,s_B)$. The training objective is
$$\min_\theta \; \mathbb{E}_{x,\epsilon}\big[\mathcal{L}(f_\theta(x;\epsilon(s)))\big] \quad \text{subject to } C(s)\le T,$$
where $f_\theta$ is the polynomial network with weights $\theta$, $\mathcal{L}$ is the task loss, and $T$ is a target latency. We alternate between two steps: training $\theta$ with noise at the current $s$, and lowering $s$ block by block.
The third component borrows its discipline from structure-based speculative compilers, and it decides where the precision boundaries go. POSH splits programs along natural structural units (loops and subroutines) rather than proving independence. It then uses a cheap profiling pass on small training inputs to prune bad tasks, which lifted its average speedup from 1.04 to 1.30. Our analogue works as follows. We take the single-entry single-exit regions that Orion already uses for bootstrap placement as the unit of precision assignment. A short profiling pass on held-out data measures how much each region's precision can drop before accuracy falls, and that measurement sets $s$.
Speculation also enters at runtime, as an optional last step. The server runs the low-precision network optimistically. The client, who alone can decrypt, checks the output's confidence margin. If the margin is below a threshold, the client requests a full-precision re-run, which is the analogue of a squash. We measure this retry rate the way TLS papers measure squash rate. The retry is a real privacy side channel, because it reveals to the server which inputs were hard. This is the same concern behind speculation defenses such as DOLMA, which enforce that speculative work is not observable. We will therefore evaluate retries only under a fixed-rate padding policy that issues a constant fraction of re-runs regardless of input. We also report the no-retry system as the headline result.
We will also test one open tension. BayesFT found that normalization layers amplify analog noise. PolyNeXt, by contrast, needs a custom BatchNorm to keep deep polynomials stable. We will measure per-block sensitivity to see whether the normalization blocks are the ones that refuse precision reduction. If they are, we test whether folding normalization into adjacent weights before noise training resolves the problem.
Baselines
The naive reference is a network trained with the HE-friendly methodology and compiled by Orion at its default, accuracy-preserving CKKS parameters. We then sweep scale bits downward with no retraining. This gives the "do nothing" accuracy-versus-latency curve. The file gives no measured version of this curve: no reported result found on Lacuna for encrypted accuracy as a function of CKKS scale bits.
The strongest existing encrypted-accuracy reference is CHET's SqueezeNet on CIFAR-10, which reached 81.5% homomorphic accuracy against an 84% plaintext baseline (CHET). The strongest latency references are EVA's 72.7 s SqueezeNet (EVA) and Orion's 8.98 hours for ResNet-50 on ImageNet, single-threaded (Orion). Our latency comparisons will re-run Orion on the same machine and thread count for every model, rather than quoting those numbers across hardware. Beating these baselines means matching the baseline network's encrypted accuracy to within 0.5 points while cutting Orion-measured latency by at least 30% through fewer bootstraps. Alternatively, it means a higher accuracy at equal latency.
The critical ablation isolates the calibrated error model. It uses the same training schedule, the same compute and the same per-block tuning budget, but injects plain zero-mean Gaussian activation noise with tuned variance instead of the measured CKKS error. A second ablation replaces the profiled per-region precision with one global scale, to isolate the structure-based assignment. A third applies BayesFT-style Bayesian optimization of per-layer noise strength, with the same number of trials as our profiling pass. This separates "calibrated noise" from "well-tuned noise". Every arm gets identical epochs, data and hyperparameter search budget.
Why not just…?
Why not just let the compiler pick smaller parameters and skip retraining? This is the most serious threat. Networks may already be robust to CKKS error well below current defaults, in which case the compiler alone captures most of the gain. The zero-retraining sweep in the Baselines settles it. If the untrained network holds accuracy down to the same scale bits as the noise-trained one, the training contribution is nil and we would say so. Our expectation is that the gap opens where bootstrap error dominates, but this is a prediction, not a result in the file.
Why not just inject ordinary Gaussian noise, as in generic robustness training? TRICE and BayesFT both report that uninformed symmetric noise underperforms tailored noise on analog devices, by 26.01% in 1st-percentile accuracy and by more than 100× on KITTI respectively. However, CKKS error might in practice look close to Gaussian. The Gaussian ablation, with a matched tuning budget, decides whether the calibration step earns its cost.
Why not just fit polynomials to a ReLU network after training? The HE-friendly methodology already reports that post-hoc approximated ReLU reached 0.7405 on CIFAR-10, against 0.872 for trained polynomials. So we build on trained polynomial networks rather than on approximations. This alternative is ruled out by existing evidence rather than by new experiments.
Why not just use a shallower network? Depth drives multiplicative depth. PolyNeXt, however, reports that polynomial networks need depth over width to train stably. We include a width-matched shallow PolyNeXt variant as an additional point on the latency-accuracy plot. If it dominates, depth reduction is the simpler answer.
Proposed Experiments
We start at small scale: MNIST LeNet-5 and SqueezeNet on CIFAR-10, compiled by EVA and Orion, where CHET and EVA give reference points. Next come the COVIDx chest X-ray sets used by the HE-friendly paper, which is the realistic private-inference case. Finally, we train a PolyNeXt-style backbone on ImageNet-1K and encrypt only a subset of validation images, since Orion's ResNet-50 takes hours per image.
For each model we report four things: encrypted Top-1 accuracy (decrypted and compared with labels), the number of bootstraps, end-to-end latency on fixed hardware, and ciphertext and key memory. For the retry variant we also report the retry rate under fixed-rate padding.
Three checks are decisive. The first is whether the noise-trained curve lies strictly above the zero-retraining curve at scale bits where the latter loses more than one point. The second is whether it also beats the tuned-Gaussian ablation. If it does not, calibration is unnecessary. The third is whether the profiled per-region precision yields more bootstrap savings than a single global scale at equal accuracy. A secondary check tests whether the normalization regions are the most precision-sensitive, which would resolve the BayesFT/PolyNeXt tension in favour of normalization being the bottleneck under encryption as well.
References
- A Methodology for Training Homomorphic Encryption Friendly Neural Networks
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- CHET: an optimizing compiler for fully-homomorphic neural-network inferencing
- EVA: An Encrypted Vector Arithmetic Language and Compiler for Efficient Homomorphic Computation
- Orion: A Fully Homomorphic Encryption Compiler for Private Deep Neural Network Inference
- A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrays
- Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators Through Training with Right-Censored Gaussian Noise
- Improving the robustness of analog deep neural networks through a Bayes-optimized noise injection approach
- POSH: a TLS compiler that exploits program structure
- DOLMA: Securing Speculation with the Principle of Transient Non-Observability