Homomorphic encryption lets a cloud server run a neural network on a client's encrypted input without ever seeing it. The catch is that the encryption scheme can only add and multiply. Every ReLU therefore has to be replaced by a polynomial, and a polynomial only tracks ReLU over a bounded input interval. Outside that interval its error grows rapidly, and the error compounds through the layers that follow. Every method that makes this replacement accurate today fine-tunes the network on its original training data. Examples include progressive replacement with coefficient tuning, "range losses" that push pre-activations into the interval, and variance penalties during retraining. That requirement fails in exactly the settings where encrypted inference matters most. A hospital or defense contractor owns a trained model but cannot release the training images to whoever is preparing the model for encrypted deployment.
We propose to do this conversion with no real data at all. Data-free compression methods already create synthetic calibration images by optimizing random noise until the network's internal statistics match those stored in its batch-normalization layers. Our central observation is that this moment matching is the wrong target for encrypted inference. Polynomial replacement fails at the tails of the pre-activation distribution, and samples that match means and variances are known to collapse toward typical, low-diversity inputs. We therefore synthesize inputs that actively seek out range excursions. These inputs stay consistent with the stored batch-norm statistics, but they are optimized to drive the current polynomial student out of its approximation interval. We then distill the original ReLU teacher into the polynomial student on these inputs. The goal is to close most of the gap to data-dependent conversion at a fixed polynomial degree, which also means a fixed encrypted latency.
Core Claim
Converting a ReLU network to a low-degree polynomial network for CKKS is mainly a range-coverage problem. CKKS is an encryption scheme that supports approximate real-number arithmetic on packed vectors. A data-free conversion succeeds or fails according to whether its synthetic inputs expose the activation magnitudes that real inputs will produce. Standard batch-norm inversion, as in ZeroQ, under-covers these tails. A synthesis objective that is adversarial in the range of the student but anchored to the teacher's stored statistics recovers them. The claim is testable. We measure the per-layer distribution of maximum pre-activation magnitude on synthetic versus held-out real data, and the rate at which real test inputs escape the polynomial interval. Both should track final encrypted accuracy.
Where This Sits
Three data-dependent methods define the conversion problem we inherit. Polynomial Adaptation of Large-Scale CNNs fine-tunes on real images with a range loss so that low-degree Remez polynomials can stand in for ReLU and GELU. SMART-PAF replaces operators one at a time with coefficient tuning and scaling, using training data at every step. ULD-Net retrains from scratch with variance penalties and a bounded normalization so that degree-2 activations stay stable. All three motivate the proposal by showing that range control is what makes low degree work. All three also assume data, which we remove. From them we reuse the range-loss form, the Remez polynomial, and the interval convention.
The data-free side supplies the synthesis machinery. ZeroQ optimizes noise to match batch-norm statistics. DSG relaxes that matching with slack margins because exact matching yields homogeneous samples. That finding constrains our design: it is the evidence that pure moment matching under-represents tails. Adversarial generator schemes such as AdaSG target the compressed model's weaknesses. SADAG shapes synthetic data toward flat loss regions. Every one of these methods optimizes for plaintext quantization error. None of them targets polynomial domain escape, which is a different failure mode: it is one-sided, concentrated in the tails, and compounds across layers.
Encryption-oriented compression work such as HE-PEx, SpENCNN, PrivCirNet, and HEQuant reshapes linear layers or bit-widths using plaintext training data. PrivCirNet and HEQuant also offload ReLU to interactive two-party protocols. We leave linear-layer co-design untouched and target the non-interactive setting those works do not address.
The research file raises a tension: inversion assumes ReLU and fixed batch-norm statistics, so swapping in polynomials might destabilize it. We resolve this by construction. Inversion always runs through the frozen ReLU teacher, whose statistics are exactly what the batch-norm layers store. Only the student carries polynomials, and its intermediate features are tied to the teacher's by feature-level distillation.
Research Program
Let $T$ be the frozen pre-trained teacher and $S_\theta$ the student. The student has the same weights at initialization, with each ReLU replaced by a fixed-degree polynomial $p$ accurate on $[-B, B]$. Let $z_l(x)$ be the student's pre-activations at layer $l$ on input $x$. We alternate two steps:
$$\max_{x}\; \lambda\, R_B(S_\theta, x) + D\big(T(x), S_\theta(x)\big) - \beta\, \mathcal{L}_{BN}(T, x), \qquad R_B(S_\theta,x)=\sum_l \operatorname{mean}\big[\max(|z_l(x)|-B,0)^2\big]$$
$$\min_{\theta}\; D\big(T(x), S_\theta(x)\big) + \sum_l \|h^T_l(x)-h^S_l(x)\|^2 + \gamma\, R_B(S_\theta, x)$$
The symbols are as follows. $R_B$ is the range-excursion penalty from range-aware training. $D$ is a KL divergence between teacher and student outputs. $h_l$ are intermediate features. $\mathcal{L}_{BN}$ is ZeroQ's batch-norm matching loss computed through the teacher. $\lambda, \beta, \gamma$ are weights.
The inner maximization finds inputs that look like training data to the teacher but push the student out of its interval or away from the teacher. The outer minimization repairs those failures. We keep a replay buffer of past synthetic batches, as in DSG's diversity argument, so the student does not oscillate between failure modes.
The final interval $B$, and with it the CKKS scale and depth, is set from the synthetic excursion distribution plus a safety margin. The program has three parts.
First, we characterize the tail-coverage gap: how far ZeroQ and DSG samples fall short of real data in per-layer maximum magnitude.
Second, we build the range-seeking conversion above and test it under real CKKS encryption using the HElayers setup that Polynomial Adaptation reports latencies with.
Third, we check whether range-seeking samples memorize training inputs more than moment-matched samples do. This matters because the deployment motivation is privacy. We will measure it with nearest-neighbor distance to the real training set, computed only for evaluation.
We scope this to batch-norm CNNs, namely ResNet-18, ResNet-20, ResNet-50, and the AlexNet variant HE-PEx uses on medical data. Transformers lack stored statistics and need softmax approximation, so we leave them out.
Baselines
The naive reference is direct post-hoc replacement: swap ReLU for the same-degree polynomial with no adaptation. The research file's discussion of SMART-PAF and ULD-Net indicates this degrades sharply. We run it with identical degree and interval to show how much any adaptation buys.
The obvious composition baseline is the one a reviewer would build in an afternoon. It generates samples with ZeroQ or DSG and then runs range-aware distillation on that fixed pool. This is the strongest data-free competitor because it differs from our method only in what the samples target. We give it the same number of synthetic images, the same number of student updates, and the same hyperparameter search budget. No reported result was found on Lacuna for data-free polynomial adaptation, so we must run this baseline ourselves. Beating it means at least a few points of top-1 accuracy at degree 2–4 on ImageNet ResNet-18, together with a measurably lower real-data escape rate.
The strongest existing method uses real data, and it serves as an upper reference rather than a competitor. Polynomial Adaptation of Large-Scale CNNs reports 76.20% top-1 for ResNet-50 on ImageNet, down from 80.86% in plaintext. ULD-Net reports 69.79% for ResNet-18 on ImageNet and 78.81% for ResNet-18 on CIFAR-100. ULD-Net trains from scratch, so it bounds what architectural redesign can achieve rather than what conversion can. We rerun range-aware training at our exact degree and interval so the gap is measured like for like. Success means recovering most of the real-data method's improvement over naive replacement, with "most" fixed in advance as at least 70%.
Two ablations isolate the key ingredient. One removes the range term from the inner maximization, leaving teacher-anchored adversarial distillation. The other removes the adversarial step entirely, leaving moment matching plus distillation. If the first ablation matches the full method, the gain comes from adversarial distillation in general rather than from range seeking, and we would report it that way.
Why not just…?
Why not just calibrate and distill on a public surrogate dataset? This is the most serious threat. CafeQ reports that external calibration data causes domain-shift losses (CafeQ), and PSAQ-ViT found its synthetic data beat limited real data at 4 bits (PSAQ-ViT). Neither result is about polynomial range. For an ImageNet model, a natural-image surrogate such as Places-365 may cover tails well. We therefore run surrogate distillation with matched budgets in two regimes: ImageNet models with Places-365, and a COVIDx CT-2A model with ImageNet as the surrogate. If the surrogate matches us on ImageNet but not on CT, we will state that the contribution is specific to the out-of-domain, air-gapped case.
Why not just set the interval from batch-norm statistics, say $B=\mu+k\sigma$ per channel, and skip synthesis? This uses no data and costs nothing. It sets the interval but cannot shrink the activations, and shrinking them is what lets a low degree work. A larger $B$ forces a higher degree or a worse approximation. We include it as a baseline at each degree. If it matches the full method at degree 8 or above, the method's value is confined to the low-degree regime, which is where the latency savings are.
Why not just reuse ZeroQ or DSG samples unchanged? This is the composition baseline above. The tail-coverage measurement settles it directly. If moment-matched samples already reproduce the real per-layer maximum magnitudes, the core claim is false.
Why not just evaluate ReLU interactively? Hybrid protocols such as those used by HEQuant and PrivCirNet avoid approximation entirely. The cost is heavy communication and a client that must stay online, as noted by A Systematic Comparison of Encrypted Machine Learning Solutions. That is a different deployment, not a competitor on the non-interactive question.
Proposed Experiments
We will evaluate on ImageNet with ResNet-18 and ResNet-50, CIFAR-100 with ResNet-20 and ResNet-18, and COVIDx CT-2A with the HE-PEx AlexNet as the regulated-domain case. At polynomial degrees 2, 4, 8, and 14, we report plaintext-simulated accuracy and accuracy under actual CKKS execution at 128-bit security. We also report multiplicative depth, bootstrapping count, and wall-clock latency. Latency should equal that of the real-data baseline at the same degree; Polynomial Adaptation reports 7 and 31 GPU minutes for ResNet-18 and ResNet-50.
The first decisive check is mechanistic. We plot, per layer, the 99.9th-percentile pre-activation magnitude for real held-out images against ZeroQ, DSG, and our samples. We also report the fraction of real test activations that fall outside $[-B,B]$. The claim requires our samples to close the tail gap and the escape rate to predict accuracy across methods.
The second check is the accuracy gap at degree 2–4 against the composition baseline and the real-data reference, with the success criteria fixed above.
The third check is the surrogate comparison in and out of domain, which determines how broad the contribution is.
A negative result on the first check, where tails are already covered by moment matching, would redirect the work toward the distillation objective rather than the synthesis.
References
- ZeroQ: A Novel Zero Shot Quantization Framework
- Polynomial Adaptation of Large-Scale CNNs for Homomorphic Encryption-Based Secure Inference
- Accurate Low-Degree Polynomial Approximation of Non-polynomial Operators for Fast Private Inference in Homomorphic Encryption
- ULD-Net: Enabling Ultra-Low-Degree Fully Polynomial Networks for Homomorphically Encrypted Inference
- Diversifying Sample Generation for Accurate Data-Free Quantization
- Sharpness-Aware Data Generation for Zero-shot Quantization
- HE-PEx: Efficient Machine Learning under Homomorphic Encryption using Pruning, Permutation and Expansion
- SpENCNN: Orchestrating Encoding and Sparsity for Fast Homomorphically Encrypted Neural Network Inference
- PrivCirNet: Efficient Private Inference via Block Circulant Transformation
- HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference
- CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
- Patch Similarity Aware Data-Free Quantization for Vision Transformers
- A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification