Spreading One Encrypted Inference Across a Cluster

Fully homomorphic encryption (FHE) lets a server run a neural network on data it can never read. The client sends encrypted inputs and gets back encrypted outputs. The cost is severe. Orion, a compiler that turns unmodified PyTorch models into encrypted programs, needs 8.98 hours for one ResNet-50 ImageNet image and 17.5 hours for one YOLO-v1 detection, running single-threaded (Orion). Compilers like Orion and Rotom have made encrypted inference much easier to write. None of them runs across more than one machine. The workflow tools domain scientists already use to reach clusters, such as Parsl, treat an encrypted program as an opaque Python task. The result is that a biologist or materials scientist who wants private inference on a sensitive image faces two walls at once: cryptographic expertise and cluster engineering.

This note proposes a compiler back end that takes one Orion-compiled encrypted inference and spreads its internal operations across the nodes of a cluster. It exposes that capability as a single Parsl-style decorator. The central bet is that distribution changes which cryptographic parameters are best, so parameters and placement must be chosen together. The most important parameter is the ring dimension, which sets how many numbers fit in one ciphertext and how large each ciphertext and key is. On one node, a large ring is attractive because it allows many multiplications between expensive noise-reset steps. On many nodes linked by a finite network, a smaller ring yields more, smaller, cheaper ciphertexts that can be processed in parallel and shipped quickly. A smaller ring also forces more frequent noise resets. We claim that the parameter set with the lowest latency shifts with node count and bandwidth. We further claim that a compiler that searches parameters and placement jointly, using a measured cost model, will cut single-inference latency below both the best single-node execution and naive distribution of a fixed single-node program.

Background and where the gap is

In CKKS-style FHE, the approximate-arithmetic lattice scheme Orion targets, every ciphertext carries noise that grows with each multiplication. When the budget runs out, the program must "bootstrap": run a heavy circuit that resets the noise (Circuit Bootstrapping: Faster and Smaller). Orion's contribution is to automate two hard steps for deep networks. It automates packing, meaning how tensor values are arranged into the slots of each ciphertext. It also places bootstraps using a shortest-path search over single-entry, single-exit subgraphs of the network. It then leaves multi-threaded, GPU and multi-node execution as future work. Rotom automates layout search for tensor programs and compiles in minutes, but it has no bootstrapping and targets a single device (Rotom). Porcupine synthesizes short BFV kernels from user sketches (Porcupine). These three papers supply the front end we reuse. The gap is that all of them stop at one device.

Two papers motivate the parameter–placement coupling. NASS shows that encrypted-inference cost is "steppy": crossing a depth or modulus threshold forces a jump to larger parameters and causes an order-of-magnitude cliff (NASS). FHEBench shows that the best scheme depends on circuit depth (FHEBench). NASS searches parameters jointly with architecture for a fixed single machine, at roughly 60 GPU-hours on CIFAR-10. We keep the architecture fixed. We search parameters jointly with the hardware layout, using a cheap analytic model instead of reinforcement learning.

Two papers distribute or split HE work, and both change the trust model. Faster Secure Data Mining via Distributed Homomorphic Encryption avoids bootstrapping by sending noisy parameters to a server that holds the secret key and re-encrypts them. It covers only linear and logistic models. SAFHIRE returns intermediate ciphertexts to the client, which decrypts, applies ReLU in plaintext, and re-encrypts (SAFHIRE). Our proposal keeps the pure non-interactive FHE setting: no party other than the client ever holds the key, and the client goes offline after sending its input. Distribution happens only among untrusted server nodes that see ciphertexts.

On the infrastructure side, Parsl supplies the programming model. It offers decorated Python functions, futures and pluggable executors. Its low-latency executor adds about 3.47 ms per task, which is negligible next to second-scale bootstrap operations. Parsl also names inter-node data staging as an open weakness (Parsl). That weakness is exactly what bites here, because FHE ciphertexts and evaluation keys are orders of magnitude larger than plaintext (A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification).

Core Claim

For a single encrypted inference on a deep branching network, the latency-optimal CKKS ring dimension and modulus chain depend on the number of nodes and the network bandwidth. A back end can find better latency than single-node execution by jointly choosing the following:

  • the parameter set,
  • Orion's bootstrap placement recomputed for that set, and
  • an assignment of ciphertext-level operations to nodes.

Its objective is a critical-path estimate of the compiled operation graph:

$$T(P, \pi) = \max_{\text{paths } p} \sum_{v \in p} \Big( c_P(v) + \mathbb{1}[\pi(v) \neq \pi(\text{pred}(v))]\, \frac{b_P}{B} \Big)$$

Here $P$ is the parameter set and $\pi$ maps each ciphertext operation $v$ to a node. The term $c_P(v)$ is the measured cost of $v$ under $P$ at its current noise level. The term $b_P$ is the byte size of a ciphertext under $P$, which is proportional to the ring dimension times the number of remaining moduli. $B$ is the link bandwidth. The indicator charges a transfer whenever an operation runs on a different node from its input. The ciphertext-size form comes directly from the scheme's representation, so the model needs no fitted transfer term. Only $c_P$ is calibrated, using FHEBench-style microbenchmarks per operation type and level.

Research Program

The system has three parts.

The first part enumerates parameter sets at 128-bit security. For each set it reruns Orion's packing and bootstrap placement. Because Orion places bootstraps for a ResNet-20 in about 2 seconds, sweeping tens of parameter sets stays within minutes.

The second part partitions the resulting ciphertext-level graph. It uses list scheduling with a communication-aware priority, the classic heuristic for scheduling task graphs with transfer costs. Bootstraps are treated as the dominant, independent-per-ciphertext tasks that anchor the schedule. Evaluation keys are replicated to every node once per deployment. Their one-time transfer is reported separately from per-inference latency, since keys amortize over many inferences.

The third part is the user surface. A scientist wraps a PyTorch model with a decorator that names a Parsl executor configuration. Ciphertexts move over a direct node-to-node channel that shares one cryptographic context per node, rather than Parsl's file staging. This targets the staging weakness Parsl itself reports.

The analysis question is when distribution helps at all. We will map the region, in node count and bandwidth, where the joint optimum beats single-node execution. We will also report where the optimal ring dimension changes.

Baselines

The naive reference is Orion as published: single-threaded CPU, its own parameter choice, no distribution. For ResNet-50 on ImageNet, Orion reports 8.98 hours, and it reports 17.5 hours for YOLO-v1 at 448×448. The file gives a relative ResNet-20 figure (2.38× over Fhelipe) but no absolute latency. We will rerun Orion on our hardware to obtain every reference number.

The strongest competitor is the same Orion program run multi-threaded on one node using all its cores. Where available, it will also run on one GPU through a GPU-capable FHE library. This is the strongest competitor because intra-node parallelism has no network cost. For pure FHE, it is the natural thing a practitioner would try first. No reported result found on Lacuna for multi-threaded or GPU pure-FHE Orion inference. SAFHIRE's A100 figure of 13.65 seconds for CIFAR-10 ResNet-20, and its 86.12× multithreaded server reduction, are for its hybrid protocol. They are not comparable to pure FHE, so we will measure this baseline ourselves.

The fairness rule is equal total cores. A 16-node cluster of 8-core machines is compared against the largest single machine with 128 cores, plus a GPU node. All systems use the same models, weights and security level. Every system, baselines included, receives the same parameter sweep budget. Beating this baseline means lower wall-clock single-inference latency at equal cores, with output error within Orion's CKKS precision, on at least ResNet-50 and YOLO-v1.

A second baseline is naive distribution. It takes Orion's single-node parameters and bootstrap placement, then assigns ciphertext operations to nodes with the same scheduler. The only thing it lacks is joint parameter search.

Ablations isolate each ingredient:

  • The fixed-parameter distributed baseline isolates the joint parameter search. This is the key test of the claim.
  • A scheduler that ignores transfer costs isolates the communication term.
  • Parsl's standard file staging versus the shared-context channel isolates the data-movement design.

Why not just…?

Why not just use a GPU or a big multi-core machine? This is the real threat. The joint search is evaluated with GPUs as well as CPU nodes, so it could still apply to multi-GPU servers. If one well-used GPU beats our cluster at equal cost on every model, the contribution shrinks. It would then cover labs that have cluster allocations but no FHE-capable GPUs, plus models whose ciphertexts and keys exceed one device's memory. The equal-core comparison and a cost-normalized comparison settle this directly. We will report the outcome either way.

Why not just run many independent encrypted inferences in parallel with Parsl? For throughput, that is correct and trivial. We will report it as the throughput ceiling. It does nothing for the latency of one inference, and the 17.5-hour YOLO-v1 case shows latency is the barrier to interactive scientific use. Our claim is restricted to single-inference latency, and the experiments state that restriction.

Why not just move non-linear layers to the client, as SAFHIRE does, or refresh noise at a key-holding server? Both avoid bootstrapping by changing who holds the key and how often parties talk. SAFHIRE requires the client to stay online through every round and to exchange large ciphertexts. It also needs shuffling and differential-privacy noise to protect the model weights. The parameter-server method trusts a key holder and has only been shown on linear models. Neither is available when the client must submit and disconnect. We therefore treat them as a different operating point, not a competitor. As context only, we will run SAFHIRE on ResNet-20 at its reported 1.25 MB/s client link.

Proposed Experiments

The models are Orion's benchmarks: ResNet-20 on CIFAR-10, ResNet-50 on ImageNet, and YOLO-v1 at 448×448. The cluster sweep covers 1, 4, 16 and 64 nodes. The interconnect is either 10 or 100 Gb/s, the latter throttled to emulate cloud links.

The metrics are:

  • single-inference wall-clock latency,
  • bytes moved per inference,
  • one-time key distribution cost,
  • output agreement with plaintext and top-1 accuracy,
  • compile time, and
  • scaling efficiency relative to the equal-core single-node baseline.

For usability, we count the lines of user code and configuration needed to go from a PyTorch model to a cluster run. We compare this count against hand-written Parsl orchestration of Orion's output.

Three checks decide the claim.

The first asks whether the latency-optimal ring dimension actually changes with node count and bandwidth. If it never moves, the joint search is unnecessary. The work then reduces to a placement system and should be reported as such.

The second asks whether the joint system beats the fixed-parameter distributed ablation by a margin larger than run-to-run variance. This attributes the gain to the coupling.

The third asks whether, at equal cores, distribution beats the multi-threaded single node on ResNet-50 and YOLO-v1. It also asks at what bandwidth that advantage disappears. The cost model is itself validated by rank-correlating predicted against measured latency across all configurations. A poorly ranking model would undermine the search regardless of the headline numbers.