Homomorphic encryption lets a cloud server run a neural network on data it cannot read. The client encrypts its input, the server computes on the scrambled numbers, and only the client can decrypt the answer. The cost is large. Encrypted arithmetic runs orders of magnitude slower than plaintext arithmetic, and the slowdown does not follow operation counts. It depends on two coarse quantities. The first is how many ciphertexts (encrypted vectors, each packing thousands of numbers into "slots") the network touches. The second is how many sequential multiplications the deepest path contains, its "multiplicative depth," because each extra level forces larger, slower encryption parameters. Deployed models also have to keep learning as new classes and domains arrive. The usual way to do that without forgetting is to grow the network, adding neurons, channels, or modules for new data while freezing the old ones. Under encryption, growth is either free or very expensive. A new channel that fits into empty slots of a ciphertext that is already being processed costs nothing. A new channel that needs one more ciphertext, one more level of depth, or a rotation offset the server has no key for costs a great deal.
This note proposes continual learning in which every growth decision is priced in encrypted cost units rather than in parameters or floating-point operations. Growth is restricted to directions that leave depth and the client's rotation-key set unchanged, and it fills existing slot vacancies before spending a new ciphertext. The claim is specific. Across a 10-task class-incremental sequence, a network grown this way should keep its measured encrypted latency nearly flat while its accuracy tracks plaintext growth methods. At matched encrypted latency, it should beat growth that is chosen by plaintext cost.
Core Claim and Where It Sits
Three findings motivate the proposal. HE-PEx prunes, permutes, and then re-inserts pruned weights into partially filled ciphertext tiles, because a half-empty tile costs the same as a full one. This lets it recover accuracy at zero added latency. That is the closest existing work, but its expansion happens only once, offline, inside fixed matrix dimensions, to undo its own pruning. It never adds capacity for new data. HEMET shows the "depth trap": mobile blocks that cut operation counts add layers, the extra depth inflates encryption parameters, and the result is slower. Its fix is a static greedy block search. Activation-Free Backbones builds polynomial-only vision models, reaching 82.7% ImageNet top-1 for its fully polynomial variant. It argues for scaling depth over width, since $L$ multiplicative layers give polynomial degree $2^L$. It never runs under encryption.
The tension between HEMET and PolyNeXt drives the design. Depth buys polynomial expressiveness, and depth is exactly what encryption punishes. Continual growth therefore has to be lateral, adding width and parallel branches at fixed depth. We test whether lateral-only growth is enough for incremental tasks, rather than assuming it.
The accuracy-cost lens comes from scientific ML. FlashSchNet showed that molecular graph network simulation is limited by memory traffic rather than arithmetic, and won a 6.5× speedup by optimizing the true bottleneck. The analogous point here is that encrypted cost is a step function of ciphertext count, depth level, and the rotation-key set. A growth policy that minimizes parameters or floating-point operations optimizes the wrong quantity.
Two growth methods supply our machinery. From MaRS we reuse a statistically grounded trigger: Wilson-score confidence intervals on routing confidence decide when new capacity is warranted, and MaRS grows from 32 to 49 slots on Tiny-ImageNet without runaway bloat. From ADEPT we reuse function-preserving initialization, so that grown units start as no-ops. Neither method accounts for slot layout.
Research Program
The backbone is a compact activation-free network in the PolyNeXt style. It uses Hadamard-product multilayer perceptrons and polynomial convolutions, sized for CIFAR and Tiny-ImageNet. It runs under the CKKS scheme (an approximate-arithmetic homomorphic scheme) at 128-bit security, in leveled mode with no bootstrapping. Depth is fixed at deployment.
The first component is a cost model. We first pack the deployed network into ciphertext tiles. Each layer then has three resources: a vacancy map (free slots inside tiles already evaluated), a fixed depth budget, and a set of rotation offsets for which the client has already uploaded Galois keys (the key material the server needs to shift slots inside a ciphertext). A candidate growth action $a$ might add $k$ channels to a layer, add a parallel branch at the same depth, or extend the output layer with new classes. Its encrypted cost is
$$\Delta L(a) = c_{\text{tile}} \cdot n_{\text{new}}(a) + c_{\text{rot}} \cdot r_{\text{new}}(a) + \infty \cdot \mathbb{1}[\text{depth increases}],$$
The terms are defined as follows. $n_{\text{new}}$ is the number of additional ciphertexts the action requires. $r_{\text{new}}$ is the number of added rotations. $c_{\text{tile}}$ and $c_{\text{rot}}$ are per-tile and per-rotation latencies measured once on the target hardware. We also forbid actions that need rotation offsets outside the existing key set, so the client never has to re-upload keys.
The second component is a layout-constrained placement step. New channels are assigned to vacant slots whose positions line up with rotations the existing schedule already performs. We solve this as a small assignment problem using the co-permutation idea from HE-PEx, but applied to grown units instead of re-inserted weights. Old units stay frozen in place.
The third component is the growth policy. When the Wilson-interval trigger fires on a new task, the policy greedily picks actions by estimated accuracy gain per unit of $\Delta L$. We estimate gain with a short probe fine-tune. Zero-cost vacancy actions are used first. Actions that spend a ciphertext are taken only under a user-set latency budget.
We then analyze how much free vacancy real packings leave. This varies with layer widths, ring dimension, and tile shape, and it determines how many tasks the method can absorb "for free" before it must pay. We report a measured curve of accuracy against cumulative encrypted latency over the task sequence.
Baselines
The naive reference is a static polynomial network trained jointly on all tasks seen so far and retrained from scratch at each arrival. It is the accuracy ceiling for this backbone. Its training cost is the price of not growing, and its encrypted latency stays constant. A second naive reference is sequential fine-tuning without growth, which bounds forgetting from the other side.
The strongest continual-learning competitor is MaRS run in plaintext with its published settings. It reports 57.50% on CIFAR-100 and 49.46% on Tiny-ImageNet with about 0.2M trainable parameters. These numbers come from MaRS, whose setup rests on a large pretrained frozen backbone. Our polynomial backbone will not match those absolute numbers, so we also re-run MaRS-style slot growth on our own backbone with identical data, task splits, and tuning budget. We then encrypt the result as-is. No reported result was found on Lacuna for any continual learning method evaluated under homomorphic encryption.
The strongest encryption-efficiency competitors are two. The first is HE-PEx applied to the final grown network, which tests whether post-hoc pruning, permutation, and re-expansion closes the gap. The second is HEMET-style static block search on the final task set. For encrypted latency the reference point is HEMET's reported 29.6s for SqueezeNet on CIFAR-10, down from 72.7s (HEMET). We will run every model through the same CKKS library, hardware, and security level.
Three ablations isolate the key ingredient. In the first, the policy uses floating-point cost instead of $\Delta L$. In the second, growth is allowed to add depth. In the third, the network is padded to tile boundaries at deployment and the padded units are used as a frozen-mask reserve, with no adaptive policy. Beating these baselines means one of two outcomes. At matched measured encrypted latency, average incremental accuracy should be higher than every encrypted baseline. Alternatively, accuracy should come within 1 point of MaRS-style growth on the same backbone at strictly lower encrypted latency and with no new rotation keys.
Why not just…?
Why not just run a static search like NASS once per new task? NASS reaches 84.6% on CIFAR-10 at half the latency of Gazelle. It needs about 60 GPU hours per search, however, and it retrains from scratch whenever the topology changes. That cost recurs at every task, and it gives no protection against forgetting. Our joint-retrain baseline measures the accuracy side of this trade-off, and we report its cumulative training compute against our method's compute.
Why not just freeze the polynomial backbone and add a linear head per task? Linear heads add no depth and are cheap, so this is a serious threat. MaRS itself notes that frozen backbones fail when their features lack the resolution the new data needs. This baseline is included directly. If it matches lateral growth on Split CIFAR-100 and Tiny-ImageNet, then backbone growth is unnecessary at this scale, and we would report that as the finding.
Why not just pad every layer to the next tile boundary and reserve the padding for future tasks? This is the most serious alternative. Free vacancy and pre-padding are nearly equivalent in cost. If the pad-and-mask ablation matches the full method, then the contribution reduces to the cost accounting, the key-set constraint, and the vacancy analysis rather than to the adaptive policy. The experiments settle this by comparing the two once the reserve runs out, around the middle of the task sequence, where only the policy decides when spending a ciphertext is worth it.
Why not just use interactive protocols that compute exact ReLUs? Hybrid protocols such as SAFENet keep exact ReLUs, but the client must stay online for every activation layer. They also transfer gigabytes per session (Systematic Comparison). We target non-interactive settings in the tradition of CryptoNets, and we report communication explicitly so the comparison is visible.
Proposed Experiments
The main benchmarks are Split CIFAR-100 and Tiny-ImageNet in 10-task class-incremental sequences. We use encrypted CIFAR-10 and MNIST as calibration points for the cost model against the HEMET, HE-PEx, and CryptoNets regimes. Metrics are average incremental accuracy, forgetting, measured encrypted latency per inference, ciphertext count, depth, rotation count, and client key-upload size.
The first decisive check is whether encrypted latency stays within 10% of its initial value across all 10 tasks while accuracy follows the plaintext growth curve. The second is whether, at matched latency, cost-priced growth beats floating-point-priced growth. If it does not, the step-function cost structure is not decision-relevant at this scale. The third is whether lateral-only growth keeps pace with depth-allowed growth in accuracy. This directly tests the depth-versus-width tension between PolyNeXt and HEMET. The fourth is the pad-reserve comparison after the vacancies are exhausted. A negative result on that check would narrow the claim to an accounting and layout contribution.
References
- HE-PEx: Efficient Machine Learning under Homomorphic Encryption using Pruning, Permutation and Expansion
- HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- $\texttt{FlashSchNet}$: Fast and Accurate Coarse-Grained Neural Network Molecular Dynamics
- MaRS: Memory-Adaptive Routing for Reliable Capacity Expansion and Knowledge Retention
- ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
- NASS: Optimizing Secure Inference via Neural Architecture Search
- SAFENet: A Secure, Accurate and Fast Neural Network Inference
- A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification
- CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy