Suppose several clients send encrypted queries to one cloud model, and each query belongs to a different task: segmentation for one client, depth estimation for another, sentiment classification for a third. Fully homomorphic encryption (FHE) lets the server compute on data it cannot read. It is also 2 to 4 orders of magnitude slower than plaintext computation, and most of that cost comes from multiplications, which add noise that must be periodically cleared by an expensive refresh step called bootstrapping. Multi-task learning offers a way to cut the cost: one backbone shared by all tasks, with small task-specific "heads" at the end. That is exactly the rigid parameter sharing the multi-task literature has spent years trying to escape, because forcing conflicting tasks through the same weights hurts accuracy.
The argument of this note is that encryption changes the accounting that made flexible sharing attractive. If the server must not learn which task a query belongs to, it cannot run only that task's path. It has to evaluate every task's private parameters for every query, so a task-specific block costs the sum over all tasks rather than one task's share. It also cannot cheaply "skip" a block for one task, because choosing between paths on encrypted data takes an extra encrypted multiplication. We propose to learn where a multi-task network should stop sharing under this task-oblivious cost model, directly inside backbones built from encryption-native polynomial operations. We also test whether sharing structures found in ordinary ReLU/Softmax networks carry over to these polynomial backbones. We suspect they do not.
Core Claim
Under task-oblivious encrypted inference, the right multi-task architecture is a single learned split profile. For each layer it specifies how many distinct task-specific weight sets exist there. It is chosen to minimize accuracy loss subject to a cost equal to the sum of all branch costs plus a charge for bootstrapping triggered by depth. Two properties of this regime make it different from plaintext sharing search.
The first property concerns the direction of the cost. Parallel task branches add to compute but not to depth: depth equals the shared depth plus the deepest branch, not the sum of branches. Stacking layers, by contrast, adds depth and therefore bootstrapping. So under FHE, "split into wider parallel branches" and "go deeper" have opposite cost signs from what a plaintext FLOP count suggests.
The second property concerns what sharing looks like in polynomial networks. The interference profile, meaning which layers suffer from conflicting task gradients, may differ from ReLU networks. Activation-Free Backbones for Image Recognition (PolyNeXt) shows that ReLUs create zero-gradient "dead zones" that break coupling between multiplied branches. Without such dead zones, every task's gradient reaches every unit, and that could change where tasks collide.
We write the search objective as
$$\min_{\theta,\pi}\ \sum_{t=1}^{T}\mathcal{L}_t(\theta,\pi)\;+\;\lambda\Big(\sum_{\ell} k_\ell(\pi)\,c_\ell\;+\;\beta\,B(\pi)\Big).$$
Here $\theta$ are the network weights, $\pi$ is the sharing policy, and $\mathcal{L}_t$ is task $t$'s loss. The quantity $k_\ell(\pi)$ is the number of distinct weight sets at layer $\ell$: it equals 1 if the layer is shared and $T$ if it is fully task-specific. The constant $c_\ell$ is the measured encrypted cost of evaluating one copy of layer $\ell$. $B(\pi)$ is the number of bootstraps implied by the policy's multiplicative depth. Finally, $\lambda$ and $\beta$ trade accuracy against cost.
The contrast with AdaShare is the substance of the proposal. AdaShare's sparsity regularizer counts blocks each task executes, which rewards per-task skipping. Our cost counts the union of distinct weight sets, and it charges skip-and-merge operations for their extra encrypted multiplication.
Research Program
The work has three parts.
Measuring the cost model. We first build and validate the cost model. Server weights stay in plaintext and client activations are encrypted, as in Encryption-Friendly LLM Architecture. That paper reports plaintext-ciphertext matrix products running 4.58× faster than ciphertext-ciphertext products. Every linear layer is therefore a plaintext-ciphertext product. The $T$ branches that fork from a shared activation can be evaluated as one wider product against concatenated weights. Deeper branch layers become block-diagonal products. We will measure $c_\ell$ for each block type (PolyMLP, PolyAttn, cubic attention, Gaussian-kernel attention), branching factor, and ciphertext packing layout in a CKKS implementation. We will then check that the summed prediction matches end-to-end encrypted latency, using FHEBench as a reference for scheme-level costs. If predicted and measured latency diverge by more than about 20%, we will replace the analytic cost with a lookup table of measured costs.
Searching the split profile. We search the split profile in two backbone families.
In vision, the backbone is a PolyNeXt-style MetaFormer, which replaces activations and Softmax with Hadamard-product blocks. We keep its stabilization recipe of sigmoid-scale gating and multi-input skips. Because its "depth-over-width" design directly raises multiplicative depth, the $\beta$ term matters here.
In language, the backbone is a small BERT-style encoder whose attention is either the $1/\sqrt{N}$-scaled cubic attention of Rethinking Softmax or the Gaussian-kernel attention of Encryption-Friendly LLM Architecture. Both papers study only single tasks. Their multi-task behaviour is unknown.
The policy is relaxed with Gumbel-Softmax, as in AdaShare. Its search space changes from per-task select/skip to a per-layer choice among "shared", "split into task clusters", and "fully split". To initialize candidate clusters we reuse AdapterShare's gradient-conflict signal from AdapterShare. Search runs in plaintext on the server's own training data. Only inference is encrypted.
The transfer study. Here the architectural-generalization question becomes explicit. For each polynomial backbone we build a matched ReLU/GELU/Softmax counterpart with the same widths and depths. We run the same search on it and measure layer-wise gradient conflict in both networks. We then port the counterpart's split profile into the polynomial network and train it there.
AdapterShare found that encoder adapters stay shared while decoder adapters split. NDDR-CNN found that loosely related tasks need to diverge early. Neither result has been tested on activation-free networks. If ported profiles match natively searched ones, sharing structure is architecture-agnostic, which is useful and cheap. If they do not, then FHE deployment requires searching sharing in the deployed architecture.
Baselines
Oblivious hard sharing. The naive reference is hard sharing with a split only at the final layer, run on the same polynomial backbone. Under task-oblivious execution it is the cheapest possible multi-task model, because only the heads are replicated. It gets the same training data, schedule, and hyperparameter budget as our method. Beating it means achieving higher average task performance at no more than 1.2× its measured encrypted latency. Alternatively, we can match its accuracy at lower latency by trading depth for width.
Isolated single-task models. The accuracy-oriented reference is $T$ independent polynomial models, one per task. Each is trained with the same total compute divided evenly across tasks and evaluated obliviously, which costs $T$ full models per query. We expect this to be the most accurate and the slowest. Our target is to come within one point of its average metric at a fraction of its latency.
Cost-blind adaptive sharing. The strongest existing sharing method on vision is AdaShare, which we rerun on the polynomial backbone with its original sparsity and sharing losses. Using the same backbone, not AdaShare's ResNet, keeps the comparison fair. We then deploy its learned policy obliviously and measure real latency. The file gives AdaShare's own reported results: a +1.1% relative gain with 80% fewer parameters than the multi-task baseline on Tiny-Taskonomy, and wins on 4 of 7 metrics on NYU v2 with 50% fewer parameters than soft-sharing methods. No absolute NYU v2 numbers for AdaShare are recorded in the file.
On language, the strongest sharing method is AdapterShare, which reports +2.33 GLUE points over AdapterFusion on BERT-large. The only encrypted multi-layer reference is Encryption-Friendly LLM Architecture, at 0.6621 average GLUE encrypted versus 0.7068 for plaintext full fine-tuning on a 2-layer model. That paper handles one task at a time. We will run it per task and obliviously as a multi-task comparison. Beating these methods means lying strictly above their accuracy–latency curve at matched parameter budgets.
Ablations. Two ablations isolate our ingredients. The first keeps our search space but swaps our oblivious cost for AdaShare's per-task block count, which isolates the cost term. The second sets $\beta=0$, which isolates the depth charge.
Why not just…?
Why not just hard-share everything and replicate only the heads? This is the most serious threat. If negative transfer is weak in polynomial networks, the cheapest design wins and the search is unnecessary. Nothing in the file measures multi-task interference in activation-free networks, so we cannot rule this out. The first experiment settles it directly. We will measure per-layer gradient conflict and the per-task accuracy gap between oblivious hard sharing and isolated models. If that gap is below about half a point on NYU v2 and on multi-task GLUE, we will report that finding. It would be a useful negative result in its own right, since it would say that rigid sharing is the correct choice under FHE.
Why not just run AdaShare or AdapterShare and deploy their policy? These methods optimize the wrong cost. A block that one task skips still runs for every other task, and merging a skipped path under encryption adds a multiplication level. A skip pattern that saves plaintext FLOPs can therefore increase encrypted latency. The cost-blind baseline and the cost-swap ablation measure exactly this gap.
Why not reveal the task label in plaintext and route non-obliviously? Then the server runs only one path, and dynamic routing becomes cheap again. This is a legitimate design when task identity is not sensitive, and we include it as a latency lower bound. However, Gap 5 in the file notes that routing on task identity leaks the client's objective. Our method targets the setting where that metadata matters, and the difference between the two latencies prices obliviousness explicitly.
Why not train with ReLU/Softmax and swap in polynomial approximations afterwards? PolyNeXt reports that natively polynomial training outperforms converted networks. SecureNets shows that high-degree approximations inflate multiplicative depth. We still include a converted hard-sharing model at one operating point to confirm this in the multi-task setting.
Proposed Experiments
Vision. Vision experiments use NYU v2, covering segmentation, depth, and surface normals, and Tiny-Taskonomy's five tasks. Models are PolyNeXt-S-scale and smaller, and accuracy uses each benchmark's standard metrics. Full-resolution encrypted dense prediction at 26M parameters may be impractical. We will therefore measure end-to-end encrypted latency on reduced-resolution inputs and on a held-out subset of images, and extrapolate to full scale with the validated cost model.
Language. Language experiments use multi-task GLUE on 2-, 4-, and 6-layer encoders with cubic or Gaussian-kernel attention, which lets us see whether the split profile moves as depth grows. The same backbones also run on the dialogue suite used by AdapterShare (BANK77, SAMSum, and others).
Metrics. For every configuration we report task metrics, measured encrypted latency per query, bootstrap count, multiplicative depth, and ciphertext bytes transferred.
Decisive checks. Four checks decide the outcome. First, our method's accuracy–latency frontier should dominate oblivious hard sharing, isolated models, and deployed AdaShare policies on at least one of the two benchmark families. Second, the cost-swap ablation should lose most of that advantage. Third, split profiles searched on ReLU/Softmax counterparts and ported to polynomial backbones should either match native profiles, which supports architecture-agnostic sharing, or measurably underperform them, which supports architecture-specific search. Both outcomes are publishable. Fourth, predicted cost should track measured encrypted latency closely enough to support the extrapolation to full-scale models.
References
- Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models
- Encryption-Friendly LLM Architecture
- FHEBench: Benchmarking Fully Homomorphic Encryption Schemes
- Rethinking Softmax: Self-Attention with Polynomial Activations
- AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
- AdapterShare: Task Correlation Modeling with Adapter Differentiation
- NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction
- SecureNets: Secure Inference of Deep Neural Networks on an Untrusted Cloud