Homomorphic encryption lets a server run a neural network on data it cannot read. The client encrypts its input, the server computes on the encrypted numbers, and only the client can decrypt the answer. The cost is large, and a surprising share of it is memory rather than arithmetic. To evaluate a network, the server needs "evaluation keys": public, client-specific material that lets it rotate and multiply encrypted vectors. It also needs the model weights re-encoded into the large polynomial format the scheme computes with. Rotation keys alone take 20% to 94% of total memory in compiled encrypted-inference programs (KeyMemRT). Encoding a large language model's weights for encrypted use costs about 4x the memory of the 16-bit originals (Sylph). Today each evaluator process allocates this material on its own heap, fills it by loading and converting from storage, and throws it away when finished.
This note proposes a different arrangement. Keys and encoded weights are written to disk once, already in the exact memory layout the evaluator computes on. Every evaluator process then maps those files directly into its address space, read-only. This is zero-copy sharing, meaning several processes read the same physical memory pages instead of each holding its own copy. The operating system then does three jobs for free. It lets N worker processes share one physical copy. It makes "loading" a key cost no more than a page fault. It makes "freeing" a key cost nothing, because clean, file-backed pages can be dropped and re-read without being written anywhere. The compiler's knowledge of when each key is next needed becomes a set of hints to the kernel rather than explicit copies. The idea borrows the "map the database into the client" design of main-memory databases (Dalí, DataBlitz) and applies it to the one part of encrypted inference that is both huge and read-only.
Core Claim
Evaluation keys and encoded model weights are large, never modified after creation, and needed by many evaluations at once. Encoded weights are shared by every tenant of a model. A client's keys are shared by every concurrent request from that client. Evaluation keys contain no secret, so sharing them read-only across the server's processes does not weaken the client's privacy. This matters because it rules out the main safety objection to zero-copy designs, which is that one process can corrupt shared state (DataBlitz needs codeword checks for exactly this reason). With read-only mappings, any stray write faults instead of corrupting the shared copy.
The claim has three parts. First, a file-backed key store in evaluator-native layout makes host memory for N concurrent evaluators grow roughly as one shared copy plus N small per-request working sets. Per-process heaps grow as N full copies. Second, cold-start time-to-first-inference falls from load-and-convert time to page-fault time. Third, when the host has less memory than the full key set, turning KeyMemRT's liveness analysis into kernel hints (prefetch ahead, drop after last use) matches or beats KeyMemRT's explicit load and clear operations on latency, while using less peak memory. That comparison is the one that could fail.
Relation to Prior Work
KeyMemRT supplies the method this work builds on. It is a compiler dialect that inserts key-load and key-clear operations from liveness analysis. Its runtime has two modes: a low-memory mode that loads keys only on demand, and a balanced mode that prefetches keys to overlap loading with computation. It reports a 1.74x geomean memory reduction over ANT-ACE and a 1.73x geomean speedup over Fhelipe.
KeyMemRT works inside one process with ordinary allocation. Every load copies bytes into a private buffer, and nothing is shared across processes. This proposal keeps KeyMemRT's liveness analysis and replaces its data path. We do not claim a better scheduling policy. We claim a cheaper and shareable mechanism under the same policy.
Three hardware designs motivate the focus on memory traffic but are not competitors on commodity machines. FHEmem places compute units next to DRAM arrays inside high-bandwidth memory stacks. APACHE puts adders in DRAM dies and compute logic on memory modules, citing key-switching traffic approaching 10,000 GB/s. MemFHE computes inside resistive RAM (RRAM) crossbars. All three show that moving key and ciphertext bytes is the problem, and all three require silicon that does not exist commercially. The SoK on FHE accelerators reaches the same conclusion across platforms: memory access to precomputed parameters and rotation keys is the main limiter.
Theodosian constrains the design. It shows that GPU key switching is already bound by device-memory bandwidth and reports CKKS bootstrapping at 12.8 ms on an RTX 5090. Streaming keys on demand across PCIe from a host mapping would therefore likely be slower, not faster. For this reason the core system targets CPU evaluators, and the GPU case is treated only as a measured side question.
The zero-copy systems supply the mechanism and set expectations for its size. Bindox maps sender and receiver buffers to the same physical pages across containers, cutting memory overhead to about 1x the payload. Sample Factory keeps tensors in shared memory and passes only indices between processes. Streamflex uses typed memory regions to keep a managed runtime from copying and garbage-collecting large buffers.
None of these systems handles data whose lifetime is predicted ahead of time by a compiler, as FHE keys are. We also deliberately do not target ciphertext handoff between processes, for example in hybrid designs like SAFHIRE. There, a ciphertext copy at memory speed is tiny next to the homomorphic operations around it, and the dominant cost is wide-area network transfer, which shared memory cannot remove.
Research Program
The first artifact is a key-store format. Every rotation key, relinearization key, and pre-encoded weight plaintext is written as page-aligned arrays in the evaluator's native representation, with residue limbs already in transform (NTT) form. An index maps each key identifier to a file offset. A modified KeyMemRT runtime then turns its compiler-inserted operations into kernel calls instead of copies. A load becomes a pointer into the mapping plus a prefetch hint. A clear becomes a "drop these pages" hint. No bytes are copied into a private heap. Worker processes, whether processes, containers, or a Python-orchestrated pool, map the same files read-only, so the kernel's page cache holds one physical copy.
The second part is analysis. We build a simple model of host memory against the number of concurrent workers N, and fit it to measured proportional set size (memory charged per process when pages are shared). We then study behavior under a memory cap below the full key set. This is where the kernel's least-recently-used eviction competes with compiler-directed hints. Without hints we expect thrashing, because FHE key access follows the program order of rotations rather than recency. Without hints the kernel may evict a key the next layer needs, and whether the hints fix this is the central empirical question.
The third part is a limited GPU probe. Keys are mapped as pinned host memory the GPU can read directly, and we measure how far this falls behind device-resident keys. This tells practitioners whether sharing extends past the CPU at all.
Baselines
The naive reference point is what most deployments do today: each worker process deserializes the full key set and encoded weights into its own heap at startup and keeps everything resident. This fixes the ceiling on memory use and the floor on steady-state latency. It is run on the same host, with the same FHE library and compiled programs, at each N.
The strongest existing method is KeyMemRT, in both its low-memory and balanced modes, one instance per worker. It is the closest competitor because it already reduces key memory and hides load latency with prefetching. Any gain from our system must come from the mechanism, not from a smarter schedule. To keep the comparison fair, both systems consume the identical liveness schedule from the kmrt dialect, on the same benchmarks, library, thread count, and memory cap. KeyMemRT's reported figures, a 1.74x geomean memory reduction over ANT-ACE and a 1.73x geomean speedup over Fhelipe, are the published reference. If its code is unavailable, we reimplement its two runtime modes on our library and first check that we recover memory reductions of roughly that size over a load-everything baseline.
Beating KeyMemRT means three things. At N=1, steady-state latency is within 5% of KeyMemRT balanced mode. At N≥4, total host memory is lower by a factor that grows with N. Under a cap below the full key set, peak memory and latency are both no worse than low-memory mode.
A same-process, thread-pool baseline shares one heap copy across threads. It isolates the question of whether cross-process mapping adds anything beyond ordinary sharing.
Two ablations separate the contributions. The first maps the library's ordinary serialized format instead of native layout. This isolates the value of compute-ready layout, because deserialization cost returns. The second maps native layout but gives the kernel no hints. This isolates the value of carrying compiler liveness information into the operating system.
Why not just…?
Why not just run the evaluators as threads in one process? Threads share one copy with no operating-system tricks, and for a single tenant on a single long-lived server this is a real threat. The thread-pool baseline measures it directly. We expect it to match our memory use at steady state. Where it should fall behind is in the settings that force separate processes: per-request crash and fault isolation between tenants, Python orchestration held back by the interpreter lock, containerized or serverless workers, and restarts. In all of these the shared heap disappears and must be rebuilt, while a file-backed store survives. If cold start and isolation turn out not to matter in the serving traces we test, we will report that the thread pool is sufficient.
Why not just use KeyMemRT's balanced mode, which already hides key loading behind computation? Prefetching hides load time but not load cost. Each worker still holds private copies and pays full conversion at cold start. How much of KeyMemRT's runtime is spent in load and clear is itself unknown: no reported result found on Lacuna for per-operation key-loading time. The first experiment below measures it. If load and clear are a negligible share of runtime at N=1, the latency claim shrinks to cold start alone, and the contribution rests on memory scaling across workers. That result would still stand, because it does not depend on load time.
Why not just buy enough memory and keep everything resident? For small convolutional networks this is correct, and we will say so. The argument weakens as models and tenant counts grow. Encoded weights multiply roughly 4x over 16-bit originals (Sylph), keys are per-client, and resident copies scale with the number of workers. The memory-versus-N curves show where on-demand mapping is required rather than merely convenient.
Why not just wait for processing-in-memory hardware? Those designs report large simulated speedups (FHEmem's 4.0x over ASICs; APACHE's 10.6x to 35.5x) on hardware that cannot be bought. Our system addresses footprint and sharing on existing servers and does not compete with their arithmetic throughput.
Proposed Experiments
The workloads are the compiled programs KeyMemRT evaluates: MLP, LoLa, LeNet on MNIST, ResNet variants on CIFAR-10 and ImageNet, and AlexNet. We add one large encoded-weight workload, a single transformer block encrypted in the style of Sylph, to stress weight sharing. The serving load is a closed-loop request generator with N ∈ {1, 2, 4, 8, 16} workers, split between single-tenant and multi-tenant mixes. In the multi-tenant mix, encoded weights are shared across tenants but each tenant's keys are separate. Metrics are total host memory (sum of proportional set sizes), per-request latency at the median and 99th percentile, time-to-first-inference after a cold start, and major page faults per inference.
The first decisive check is the runtime decomposition of KeyMemRT at N=1. It tells us whether the latency claim is live or only the memory claim. The second is the memory-versus-N curve. The claim holds if our slope is near the per-request working set while both baselines grow by close to a full key set per worker. The third is a sweep of a memory cap from 100% down to 25% of the full key set. At each level we compare the hinted mapping, the unhinted mapping, and KeyMemRT's low-memory mode. If the hinted mapping loses to KeyMemRT at tight caps, we conclude that explicit copying into controlled buffers beats the kernel's page cache for this access pattern, and we report it as a negative result. The final check is the GPU probe, which reports the latency ratio of host-mapped to device-resident keys for one key-switch-heavy layer.
References
- KeyMemRT Compiler and Runtime: Unlocking Memory-Scalable FHE
- Scaling up Privacy-Preserving ML: A CKKS Implementation of Llama-2-7B
- Dalí: A High Performance Main Memory Storage Manager
- DataBlitz: A High Performance Main-Memory Storage Manager
- FHEmem: A Processing In-Memory Accelerator for Fully Homomorphic Encryption
- APACHE: A Processing-Near-Memory Architecture for Multi-Scheme Fully Homomorphic Encryption
- MemFHE: End-to-end Computing with Fully Homomorphic Encryption in Memory
- SoK: Fully Homomorphic Encryption Accelerators
- Theodosian: A Deep Dive into Memory-Hierarchy-Centric FHE Acceleration
- Bindox: An Efficient and Secure Cross-System IPC Mechanism for Multi-Platform Containers
- Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning
- Streamflex: high-throughput stream programming in java
- Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption