Refresh-and-Requantize: Placing Encrypted GPU Segments Between Enclave Visits

Private inference has two families of tools that rarely meet in one system. Homomorphic encryption lets an untrusted server run a neural network directly on encrypted inputs. It is mathematically airtight but slow: hours per ImageNet image on a CPU and seconds on a GPU. Most of that cost comes from approximating nonlinear functions with polynomials and from periodically "refreshing" ciphertext noise. A trusted execution environment, or enclave (a hardware-isolated region of a CPU whose memory the host operating system cannot read), runs ordinary code at near-native speed, but it has little protected memory and no GPU. We propose a compiler that takes one ordinary PyTorch model and decides, layer segment by layer segment, where to run it. Options are inside the enclave, on the host GPU under encryption, or on the host GPU under one-time-pad masking. The enclave is used for three jobs encryption does badly: resetting noise, computing exact nonlinearities, and rounding activations back to very low bit-widths.

The point that is specific to this proposal is what happens at each enclave visit. If every visit decrypts, applies the exact activation, re-quantizes to 2–4 bits, and re-encrypts, then the encrypted segments in between only ever carry small integers. That lets the encryption scheme use a tiny plaintext modulus and cheaper parameters. Ultra-low-bit quantization therefore stops being only a source of accuracy loss and becomes the knob that sets the cost of encrypted computation. We also commit to the honest comparison. Encryption here adds no confidentiality beyond what the enclave already provides. It is purely a way to move linear algebra onto a GPU while data stays sealed. So it must beat the cheaper ways of doing that, and the experiments are designed so it can lose.

Core Claim

Consider segments of linear layers evaluated in exact integer homomorphic arithmetic (BFV-style schemes, which compute modulo a plaintext modulus $p$). For an inner product of length $n$ between $b_w$-bit weights and $b_a$-bit activations, correctness requires roughly

$$\log_2 p \;\ge\; b_w + b_a + \log_2 n + 1,$$

because the worst-case accumulated sum must not wrap around modulo $p$. Chaining $k$ linear operations without rounding multiplies this growth. If the enclave re-quantizes at every refresh, the bound resets at each visit. The compiler can then choose segment boundaries and bit-widths jointly. Our claim is that this joint choice yields a regime with two properties. First, end-to-end latency falls well below pure homomorphic inference (NeuJeans, Orion). Second, accuracy is preserved better than with polynomial-activation pipelines, because nonlinearities stay exact. We further claim that this regime is exactly where the model's quantized weights no longer fit in enclave memory and masking-based offload carries large per-query precomputation. Outside that regime we expect simpler designs to win, and we will report that.

This reframes the "saturation barrier" in HEQuant. HEQuant reports that, in two-party protocols, the communication gains from quantization plateau below 4 bits because ciphertext parameters cannot shrink further. HEQuant has no exact rounding step between layers, so accumulators grow unchecked. Enclave refresh supplies that step, and we test whether it moves the plateau.

Relation to Prior Work

Several systems already pair enclaves with homomorphic encryption, and they supply the mechanism we reuse. TEEFHE moves bootstrapping (homomorphic noise reset) into an SGX enclave. It reports noise removal 4 to 6 orders of magnitude faster than software, and memory reduced from 50.02 GiB to 225.86 MiB on iDASH logistic regression. It also hardens the enclave with data-independent conditional moves, which we adopt. Two further systems use the enclave only as a key and re-encryption manager on MNIST-scale CNNs: Liu & Zhang's outsourced-inference scheme, which uses square activations, and Confidential Model Refining. Drucker & Gueron combine additive-only encryption with an enclave for integrity. None of these quantize, compile from a general model, or evaluate beyond logistic regression or MNIST.

The closest design is SAFHIRE. It runs linear layers encrypted on the server and sends intermediate results to the client, which applies exact ReLU and re-encrypts. It uses 8–16-bit quantization-aware training and reports 1.5×–10.5× lower latency than Orion. Replacing SAFHIRE's client with a server-side enclave is the obvious idea at this intersection. We treat it as an ablation, not our contribution. What we add is threefold: rounding to sub-4-bit at each visit, choosing visit frequency per segment rather than after every layer, and a cost model that also considers non-cryptographic placements.

For the model interface, we build on Orion. Orion compiles PyTorch graphs to homomorphic circuits and places bootstraps by shortest-path search over single-entry single-exit subgraphs. We replace "bootstrap" nodes with "enclave refresh" nodes that carry a different cost and may host non-polynomial operations. Packing decisions reuse the layout reasoning of Rotom. Rotom assumes leveled encryption without bootstrapping, which is exactly the setting our segments live in.

The quantization side supplies both the method and a constraint. QuIP shows that 2-bit weights work when weights are first rotated by random orthogonal matrices. It reports WikiText2 perplexity of 6.33 on Llama 2 70B, against 123.91 for OPTQ. CRVQ shows that roughly 2% of channels need extra precision near 1 bit. Neither has a lowering path to encrypted slots. The weight-side rotation can be folded offline. The activation-side rotation cannot cross a nonlinearity, so we perform it in plaintext inside the enclave at refresh time. CRVQ's high-precision channels become a per-channel bit-width the compiler must accommodate in its modulus bound.

Two results constrain the design. TEESLICE shows that naive enclave/GPU partitions leak weights and membership. It also shows that one-time-pad masked GPU offload with partition-before-training keeps only 4.95% of FLOPs in the enclave and runs 18.37× faster than whole-model shielding. This is a strong competitor that needs no homomorphic encryption at all. PRESAGE reports enclaves running 120× faster than homomorphic encryption on genomics, so plain enclave execution must be a baseline.

Research Program

We will build a compiler front end that ingests a PyTorch model plus a quantization recipe. The recipe specifies per-layer weight and activation bit-widths, optional incoherence rotation, and optional CRVQ-style high-precision channels. The compiler lowers the model into an intermediate representation with three placement types for each subgraph:

  • inside the enclave in plaintext;
  • on the host GPU under exact homomorphic arithmetic;
  • on the host GPU under one-time-pad masking with precomputed unmasking terms.

Each placement has a measured cost model. Homomorphic segments are costed by kernel benchmarks as a function of ring dimension, modulus chain, and rotations. Enclave segments are costed by compute plus a paging penalty once the working set exceeds protected memory. Masked segments are costed by online GPU time plus the offline precomputation and its storage, amortized per query. Transfers are costed by ciphertext or masked-tensor size over the enclave boundary.

Placement is a shortest-path problem over the single-entry single-exit decomposition, as in Orion. One added constraint is the modulus bound above, which determines the cheapest encryption parameters each candidate segment can use. The enclave runtime implements refresh-and-requantize in constant-time code. That means decryption, exact ReLU, GELU or softmax, plaintext Hadamard rotation where QuIP-style incoherence is used, rounding to the target bit-width, and re-encryption.

The analysis part is a crossover map. It records which placement wins as a function of model size relative to enclave memory, bit-width, network depth, and query volume (since volume governs the amortization of masking precomputation).

Baselines

The do-nothing references are plaintext GPU inference, which offers no privacy and gives the latency floor and accuracy ceiling, and full-precision whole-model enclave execution with paging. These bracket every result.

The strongest pure-encryption competitor is NeuJeans. It fuses convolution with bootstrapping, reports 5.35 seconds for ResNet18 on ImageNet and 56.08 seconds for ResNet50 on an A100, reports 66.4% top-1 for ResNet18-AESPA, and notes a 2.3–2.4% accuracy drop from polynomial approximations. We will compare on the same GPU class and the same ImageNet validation split. Beating it means lower end-to-end latency at equal or higher top-1. We also report Orion at its published setting (ResNet-50 in 8.98 hours single-threaded) as the compiler reference, rerun multithreaded on our hardware.

The strongest hybrid competitor is SAFHIRE, which reports 13.65 seconds for ResNet-20 on CIFAR-10 on an A100 and accuracy within 2.1% of Orion. We run it with our enclave standing in for its client. This removes its network round trips and makes the comparison favorable to SAFHIRE. It uses the same quantization-aware training budget as our models, so any gain must come from low-bit requantization and adaptive refresh placement rather than from removing the client.

The strongest non-cryptographic competitors are two. The first is TEESLICE-style masked offload with its reported 18.37× speedup over whole-model shielding. We retrain under its partition-before-training recipe with equal training compute and charge its offline precomputation to the per-query cost. The second is whole-model enclave execution of the same low-bit quantized network (no reported result found on Lacuna for quantized whole-model enclave inference on these benchmarks). We must beat both in at least one clearly characterized regime. If we do not, that is the finding.

Three ablations isolate the key ingredient. The first is enclave refresh without requantization, with activations kept at 8–16 bits; this is the TEEFHE-into-Orion combination. The second is requantization without adaptive placement, with a refresh after every nonlinearity. The third is low-bit models without incoherence rotation, which tests whether QuIP's activation rotation at refresh is needed for accuracy.

Why not just…?

Why not just quantize the model to 2–3 bits and run the whole thing inside the enclave? This is the most serious threat. A 2-bit ResNet-50 has roughly 6 MB of weights and fits comfortably in a 128 MB enclave page cache. For CNNs at ImageNet scale we expect this baseline to win, and we will report that. The proposal's claim is limited to models whose quantized weights and activations exceed protected memory, or whose FLOP count makes CPU-only execution slower than encrypted GPU segments. The crossover experiment over model size directly locates that boundary. If no such regime exists for current enclave memory sizes, the system reduces to "quantize and enclave," and we say so.

Why not just mask activations with one-time pads and run linear layers in plaintext on the GPU, as TEESLICE does? Masking is far cheaper per online operation than homomorphic arithmetic. Its costs are elsewhere: precomputing unmasking terms for fresh masks per query, plus the security caveats TEESLICE itself raises about long-horizon mask management. Our cost model charges those costs explicitly. We expect masking to win at low query volume with offline slack, and homomorphic segments to win when segments are long and precomputation cannot be amortized. If masking wins everywhere once offline costs are fairly counted, homomorphic encryption has no role in this setting, and that negative result is worth publishing.

Why not just put SAFHIRE's client logic in an enclave and keep 8-bit precision? That is our first ablation. It already removes SAFHIRE's dependence on an online client. What it does not do is shrink the plaintext modulus, because 8–16-bit activations accumulate wide sums. The modulus bound predicts the parameter savings from 3-bit requantization. If measured latency at equal accuracy is not meaningfully lower than with 8-bit refresh, low-bit quantization contributes nothing here and HEQuant's saturation barrier holds even with exact rounding.

Why not just use better polynomial activations and stay fully homomorphic? NeuJeans already shows the price: a 2.3–2.4% accuracy drop on ImageNet, plus bootstrapping that dominates latency. Activation-free polynomial backbones such as the MetaFormer polynomial alternatives require retraining a new architecture. Our comparison against NeuJeans at matched top-1 settles whether exact enclave nonlinearities buy accuracy, speed, or both. This alternative is the right choice whenever no enclave can be trusted, which is a different threat model.

Proposed Experiments

The vision track uses ResNet-20 on CIFAR-10, ResNet-18 and ResNet-50 on ImageNet, and ResNet-18 on Tiny ImageNet. Models are quantized to weight and activation bit-widths from 8 down to 2. Each bit-width is run with and without incoherence rotation and with CRVQ-style high-precision channels. We report top-1 accuracy, end-to-end latency, bytes crossing the enclave boundary, enclave peak memory, and amortized offline cost, all on one A100 host with an SGX-capable CPU.

The transformer track uses a BERT-base encoder on GLUE SST-2, MRPC and RTE, the GLUE subset TEESLICE used (no reported number found on Lacuna for TEESLICE's GLUE accuracy). This track tests softmax and GELU at refresh points and models whose size exceeds protected memory. A separate WikiText2 perplexity check confirms that our activation-quantization recipe does not reintroduce the low-bit collapse that QuIP avoids for weights.

Three outcomes are decisive. First, at matched accuracy, does requantize-at-refresh lower encrypted-segment latency relative to 8-bit refresh by roughly what the modulus bound predicts? A large mismatch would mean parameter cost is not governed by accumulator width. Second, on the size sweep, is there a model scale above which our placement beats both quantized whole-enclave execution and masked offload with offline costs included? Third, does exact enclave nonlinearity recover the 2.3–2.4% polynomial-approximation gap NeuJeans reports? A security sanity check accompanies all three. We rerun TEESLICE's model-stealing and membership-inference attacks against every partition the compiler emits, and verify that refresh code passes constant-time tests under TEEFHE's conditional-move discipline.