Running a neural network on encrypted data with fully homomorphic encryption (FHE), a cryptographic scheme in which a server computes directly on ciphertexts it cannot read, is expensive. The main driver of the cost is not the number of operations. It is multiplicative depth, the longest chain of multiplications any value passes through. Each multiplication adds noise to the ciphertext. Deeper chains therefore need larger cryptographic parameters, and past a point they need bootstrapping, a very slow operation that refreshes the noise. The networks that work best under FHE contain only polynomial operations, and these networks get their expressive power from depth. Stacking multiplications raises the polynomial degree. The accuracy recipe and the cost model therefore point in opposite directions.
We propose to remove that conflict with teaching rather than search. Fix a hard multiplicative-depth budget for an activation-free polynomial student. Then train it to reproduce the intermediate representations of a deep, unconstrained network, a ConvNeXt or Vision Transformer, that will never run under encryption. The student only has to copy features the teacher has already discovered, so it can be much shallower than a polynomial network trained from scratch. The bet is that cross-architecture feature alignment moves the accuracy-versus-depth curve of polynomial networks. If it does, the gain turns directly into fewer ciphertext levels, smaller encryption parameters, and fewer or no bootstraps.
Core Claim
At a fixed multiplicative-depth budget, a fully polynomial student trained with intermediate-feature alignment to a deep non-polynomial teacher will reach accuracy that from-scratch polynomial networks only reach at substantially greater depth. A meaningful part of that gain will come from aligning intermediate features, not only from matching the teacher's output distribution. A second, testable claim is that the alignment signal stabilizes training. It should let us drop the non-polynomial stabilizers and multi-stage schedules that current polynomial backbones depend on.
The tension is documented in the file. HEMET shows that architectures designed to minimise operations can run slower under leveled FHE because they add depth. That is the "multiplicative depth trap." HEMET answers with a greedy block search that never uses a teacher. PolyNeXt builds activation-free MetaFormer backbones from Hadamard products and a polynomial replacement for softmax attention. It explicitly favours depth over width to raise polynomial degree. It trains from scratch, uses sigmoid gating and staged recipes for stability, and was never run in an encrypted runtime. Our proposal takes PolyNeXt's building blocks as the student's vocabulary and HEMET's depth accounting as the cost model. The new ingredient is supervision that neither paper uses.
How This Differs from Existing HE-Friendly Distillation
Three papers already combine distillation with encryption, and each stops short of this question. Baruch et al. replace ReLUs with learnable quadratics, using a smooth transition and logit-level distillation from the same architecture's ReLU version. Their gap to the ReLU baseline stays at 0.32–5.3% on AlexNet. Switching to average pooling costs about 8% on SqueezeNet. They never align intermediate features, never change architecture between teacher and student, and keep the teacher's depth. THE-X distills isolated LayerNorm operators in BERT-tiny. It sends ReLUs to the client in an interactive protocol, which is exactly the round-trip cost we want to avoid. Bi-CryptoNets uses feature distillation, but between a same-family teacher and a dual-branch student that splits each image into encrypted and plaintext regions. That is not latent alignment across architecture families under a depth budget.
Our proposal therefore differs on three axes at once. The teacher and student come from different architecture families. Alignment happens at intermediate layers, not only at the output. The student's depth is a hard constraint rather than inherited from the teacher.
Research Program
Student. The student is a fully polynomial network assembled from PolyMLP, PolyConv and PolyAttn blocks (PolyNeXt), using its BatchNorm-based fully polynomial variant. The sigmoid gating is removed in our main configuration. We count multiplicative depth exactly. Each Hadamard product costs one level. The attention power $p$ costs $\lceil \log_2 p \rceil$ levels. Linear layers and BatchNorm cost nothing extra once coefficients are merged into the following weights, following HEMET's coefficient merging. We build students at budgets of roughly 8, 12, 16 and 24 levels, keeping parameter count fixed.
Teacher. The teacher is a pretrained ConvNeXt-style CNN or a DeiT-style Vision Transformer. Using both lets us test whether the teacher's architecture family matters.
Alignment. The alignment design is shaped by a constraining result. From Per-Image Low-Rank to Encoding Mismatch reports that intermediate feature distillation into Vision Transformers often fails because teacher and student encode the same information in mismatched coordinates. Plain mean-squared error between projected features is therefore a risky default. We use two losses side by side:
- Projected feature loss. A training-only linear projector on the student side, in the spirit of the $1 \times 1$ stitch layers of Deep Model Reassembly and the depth-wise aligner of DearKD, compares projected student features to teacher features.
- Relational loss. A second loss matches the token-by-token similarity (Gram) matrices of teacher and student. It does not depend on coordinates at all.
Both projectors are discarded after training, so the alignment adds nothing to the encrypted circuit. This is also why we avoid the attention-based projectors and adversarial discriminators of Cross-Architecture Knowledge Distillation. They would be fine at training time, but they add instability we do not need.
Layer assignment. A shallow student cannot copy every teacher layer. We must decide which teacher layer each student stage should imitate. We compare two rules:
- Uniform assignment. Student stage $k$ of $K$ is paired with teacher layer $\lfloor kM/K \rfloor$, where $M$ is the teacher's depth.
- Learned assignment. After a short warm-up, we compute centred kernel alignment (CKA), a similarity score between two sets of activations, between every student stage and every teacher layer, as DeRy does for grouping blocks. We then pick a monotone pairing that maximises total CKA by dynamic programming.
The hypothesis is that shallow students should skip teacher layers whose features are too high-degree to reach at that depth. The learned assignment should find these.
Encrypted measurement. Students that win in plaintext are compiled to a leveled CKKS implementation, an FHE scheme for approximate real-number arithmetic. For each we record the number of levels consumed, the ring dimension the depth requires, bootstrap count, single-image latency, and ciphertext and key sizes. This answers PolyNeXt's open question of whether these blocks work outside plaintext simulation, and it puts the depth savings into wall-clock terms.
Baselines
Naive reference. The do-nothing reference is the same depth-budgeted polynomial student trained from scratch on labels alone, with the same data, epochs, augmentation and optimizer as the distilled student. It shows what the depth cap costs without a teacher. Every other number is read against it.
Strongest existing method. This is PolyNeXt itself, trained from scratch with its full stabilization recipe at its native depth. The file reports its fully polynomial variant at 82.7% ImageNet-1K top-1, against 83.9% for PolyNeXt-S with gating (PolyNeXt). The file does not report its multiplicative depth. We will compute that from the published architecture under our counting rules, and we will retrain PolyNeXt at our reduced budgets with an equal tuning budget per configuration.
Beating this baseline means one of two things. Either a distilled student reaches 82.7% at a clearly smaller depth, with at least 25% fewer levels as our working threshold. Or it exceeds the from-scratch PolyNeXt curve at every matched depth by more than seed-to-seed variance, measured over three seeds.
Encrypted-latency reference. On CIFAR at 32×32, HEMET's searched SqueezeNet-based network runs encrypted inference at 29.6 s and its Inception-based network at 83.2 s (HEMET). We will compare against these reported latencies, and against our own HEMET reimplementation on our hardware, at matched accuracy. Wall-clock times across papers are not directly comparable, so the reimplementation is the binding comparison.
Same-architecture polynomialization. The Baruch et al. recipe converts the teacher itself into a polynomial network with learnable quadratics, a smooth transition and logit distillation. It is run on COVIDx at 224×224, where they report closing the gap to within 0.32–5.3% on AlexNet. We report its multiplicative depth next to ours.
Key ablation. The ablation that isolates our ingredient is logit-only distillation from the same cross-architecture teacher into the same depth-budgeted student, with identical compute. Intermediate alignment is the claimed contribution. If logit-only distillation closes most of the gap to our method, the claim fails. We further ablate uniform versus learned layer assignment, and projected versus relational loss.
Why not just…?
Why not just distill logits from the strong teacher and skip feature alignment? This is the most serious threat. Cross-architecture logit distillation is cheap, and Cross-Architecture Knowledge Distillation shows Transformer teachers help CNN students: an average 2.7% gain over CNN-to-CNN distillation on CIFAR-100. The indirect evidence for intermediate supervision is Baruch et al.'s persistent gap under logit-only distillation, especially after pooling changes. That evidence is not decisive. The logit-only ablation above settles it. We will report the share of the distilled-versus-scratch gain that comes from intermediate terms at each depth budget. If that share is near zero at low depth, we will report the simpler method as the answer.
Why not just search architectures under a depth constraint, as HEMET and SAFENet do? Search chooses which blocks to use. It does not supply a better training signal. In the file, HEMET gains 0.4–0.5% accuracy on 32×32 inputs, and SAFENet is demonstrated on networks up to ResNet-32. The two approaches are complementary rather than competing. To check this, we run alignment on top of a HEMET-searched CIFAR network as well as on our hand-designed students. Gains in both cases would show alignment adds something search does not.
Why not just keep the deep model and pay for bootstrapping with co-adapted approximations, as HEAT does? HEAT reduces bootstraps by 1.6x on GPT-2, but encrypted generation still takes about 37.6 s per token. This alternative accepts the depth cost rather than removing it. Our encrypted measurements report bootstrap counts directly. The question is whether a shallow aligned student avoids bootstrapping entirely at accuracy comparable to a deeper network that needs it.
Why not just use a client-aided protocol for the nonlinearities? THE-X keeps accuracy within 1.48% on GLUE this way. However, every nonlinear layer becomes a network round trip, and A Systematic Comparison of Encrypted Machine Learning Solutions documents traffic in the tens of gigabytes per deep-CNN inference. Our target is the non-interactive setting. We treat interactive protocols as a different deployment regime, not a baseline we must beat on accuracy.
Proposed Experiments
Experiment 1: accuracy versus depth. The central plot is ImageNet-1K top-1 against multiplicative depth for four curves: scratch student, logit-distilled student, feature-aligned student, and PolyNeXt retrained at matched budgets. All runs are in plaintext. The decisive check is whether the feature-aligned curve reaches PolyNeXt's 82.7% at substantially fewer levels.
Experiment 2: stability without non-polynomial gating. We train the fully polynomial student with sigmoid gating removed, with and without alignment. We count divergent runs over seeds and record the learning-rate range over which training succeeds. Alignment should widen the stable range. If it does not, the stability claim is false and we will say so.
Experiment 3: pooling. On COVIDx and COVIDx CT-2A at 224×224, we test whether intermediate alignment recovers the roughly 8% loss from average pooling that Baruch et al. report. We compare against their recipe at equal compute.
Experiment 4: encrypted runs. On CIFAR-10 and CIFAR-100 we run full encrypted inference for the best student at each budget. We report levels, ring dimension, bootstraps, latency and memory alongside the HEMET reference. The decisive outcome is a bootstrap-free student that matches HEMET's accuracy at lower latency.
Experiment 5: robustness and teacher dependence. A secondary check evaluates on ImageNet-C and ImageNet-R. This tests whether the aligned students inherit the teacher's robustness or only its clean accuracy. The same evaluation compares CNN and Vision Transformer teachers, to test whether the teacher's architecture family matters for a polynomial student.
References
- HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- A Methodology for Training Homomorphic Encryption Friendly Neural Networks
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- Bi-CryptoNets: Leveraging Different-Level Privacy for Encrypted Inference.
- From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers
- Deep Model Reassembly
- DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers
- Cross-Architecture Knowledge Distillation
- SAFENet: A Secure, Accurate and Fast Neural Network Inference
- HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
- A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification