Homomorphic encryption lets a cloud server run a neural network on data it cannot read. The price is that every rotation of an encrypted vector needs its own large "evaluation key", which is public helper material the client generates and the server must hold. Rotation keys alone take 20% to 94% of an encrypted program's memory, often tens or hundreds of gigabytes (KeyMemRT). A server that serves many clients, each with its own key set, cannot keep every key in memory, so keys will live on NVMe drives or shared file systems and be streamed in on demand. Simulation suggests this is disastrous: storage I/O slows a GPU accelerator by up to 22x and an ASIC by up to 357x (The Future of FHE from a Storage I/O Perspective). Nobody has yet built or measured a real storage path for these keys.
Our claim is that the encryption compiler already knows, before the program runs, exactly which pieces of which key are needed and when. It can therefore write the key file in the order it will be read, leave out the parts that will never be touched, and hand the runtime a fixed table of byte offsets. The runtime then issues plain sequential reads with no file lookups, no per-key opens, and no page faults. Put another way, the compiler does the job normally done by file system metadata, so that job costs nothing when the program runs. The fine-grained, level-aware key storage that saves bytes would otherwise explode into millions of small files. The static layout removes that metadata cost.
Core Claim
In CKKS, an encryption scheme for approximate arithmetic on real numbers, each ciphertext coefficient is stored as residues modulo a chain of primes. Each residue polynomial is called a "limb". As a computation proceeds, primes are dropped, so a ciphertext at level $\ell$ carries only $\ell+1$ limbs. The key-switching step behind a rotation at level $\ell$ likewise reads only the key limbs matching the active primes (plus the auxiliary primes). The key limbs for primes that have already been dropped are never touched. A compiler that tracks level can therefore turn "load rotation key $g$" into "load these specific limbs of key $g$". Storing keys at that granularity creates a very large number of small objects: many tenants, times hundreds of Galois elements, times many digits and limbs. That is exactly the regime where file system metadata dominates. Opening and closing files takes nearly 40% of I/O time in leadership HPC applications (HPC characterization), and deep learning jobs on shared parallel file systems wait on data 67% to 85% of the time (Hvac).
The proposal resolves this tension with one artifact per tenant-program pair. It is a single preallocated file whose extents hold key limbs in the order the compiled schedule consumes them, with never-used limbs omitted. Alongside it sits a compile-time offset plan that the runtime executes directly through asynchronous direct I/O into pinned host or device buffers. The file system is involved only once, at open time.
Relation to Prior Work
Three works motivate the gap without closing it. The Future of FHE from a Storage I/O Perspective uses the SimGrid simulator to show that evaluation keys drive storage demand. HELR moves 4.5x more data than ResNet-20 on the Sharp ASIC because it relies heavily on rotation keys. That study models bandwidth only, with no real file system or metadata operations, and it leaves "locality-first scheduling" as future work. We build that scheduling on real hardware.
KeyMemRT supplies our method. It is an MLIR compiler that runs liveness analysis over rotation keys, inserts load and clear operations, and prefetches keys in a "balanced" mode. It treats a load as an in-memory operation at whole-key granularity on one node. We extend its analysis in two ways: we add level information so it can project keys down to the limbs actually needed, and we compile its schedule down to a storage layout.
Hvac shows that computing file locations by hashing, instead of asking a metadata server, cuts training time by 25% over GPFS. It is built for static plaintext datasets. We push its idea one step further: the location is computed at compile time rather than hashed at run time.
Two systems papers constrain the design. SplitFS shows that user-space data paths still pay kernel costs on every open and every extent change, and that failing to get huge pages costs up to 50% for memory-mapped access. This is why we avoid both per-object files and mmap-driven page faults. An Empirical Evaluation of Columnar Storage Formats finds two relevant things. Parsing a monolithic metadata footer can take longer than reading the data itself. Zstd compression on NVMe costs up to 4.2x over uncompressed scans. Together these argue for a tiny precomputed offset plan over a self-describing format, and against compressing keys.
FLSSM shards encrypted models across aggregators but treats them as abstract vectors, never as stored files.
Research Program
We will build three things. The first is a level-aware extension to a KeyMemRT-style liveness pass. It emits, for each key-switch, the tuple of Galois element, level, and required digit and limb ranges. The second is a layout generator. It orders these limb blocks by first use, aligns them to the device block size and to GPU transfer granularity, merges adjacent reads, and records each read as an offset and length. Blocks reused later in the program are placed at their first use and cached according to the liveness intervals. The third is a small user-space runtime. It opens the key file once, submits the plan through asynchronous direct I/O with a lookahead window matched to measured compute time, and places blocks into pinned buffers for transfer to the accelerator.
For the multi-tenant case, tenant files are laid out with contiguous extents on node-local NVMe. When they sit on a shared parallel file system, each tenant costs one open per job rather than one per key or limb. We will also measure how many metadata-server operations each design generates as the number of concurrent tenants grows. This is the "thundering herd" condition that destabilized shared metadata servers in BOSS-LDG.
Baselines
The naive reference stores one file per rotation key on local NVMe and loads each key with an open and read on demand, with no prefetching. It is what a straightforward port of an in-memory library to disk would do. It is run on the same GPU, CKKS parameters, and programs. It shows how large the problem is in practice, and it gives the first real-hardware counterpart to the simulated slowdowns. The simulated slowdown for ResNet-20 on TensorFHE is 3.8x over PCIe and 15.2x over RDMA (Storage I/O Perspective).
The strongest competitor is KeyMemRT's balanced mode with all keys in a single file that is memory-mapped, and separately read with pread. Huge pages are enabled and prefetch hints are passed to the operating system. This is exactly what a careful engineer would build without our work. KeyMemRT reports a 1.73x geomean speedup over Fhelipe and a 1.74x geomean memory reduction over ANT-ACE, but only for in-memory keys. We found no reported result on Lacuna for KeyMemRT, or any FHE runtime, with keys backed by NVMe or a parallel file system. We will therefore run it ourselves with an equal tuning budget: we sweep its prefetch distance and the mmap readahead settings, and report the best configuration. Beating it means lower end-to-end latency and lower stall-on-key time at a fixed resident-memory cap. On a single tenant the margin must be large enough to survive run-to-run variance. The target regime is a memory cap well below the total key size.
The third baseline is an Hvac-style distributed cache that holds whole keys on node-local NVMe with hashed placement. It represents the strongest idea from the metadata literature applied directly.
Two ablations isolate our ingredients. The first keeps the schedule-ordered layout but stores whole keys, which isolates level projection. The second stores level-projected limbs as individual files resolved at run time, which isolates the static offset plan and directly measures the metadata cost our design removes. A granularity sweep from one file per key, to one file per limb, to one file per program traces out the tradeoff between bytes read and number of metadata operations.
Why not just…?
Why not just provision enough memory to hold every key? For one tenant with a small model, this works, and our system should give no benefit there. We will report that case honestly. The target is a server multiplexing many tenants, each with a key set that can reach tens to hundreds of gigabytes (KeyMemRT). Aggregate key memory then grows with the number of tenants while compute does not. The experiments sweep the ratio of the memory cap to total key size, so the crossover point is measured rather than asserted.
Why not just put all keys in one file, mmap it, and let KeyMemRT prefetch? This is the real threat, and it is our strongest baseline. It already removes most metadata operations. What it cannot do is skip limbs that the program will never read at its current level, or order reads by schedule rather than by key identifier. It also routes data through page faults, which SplitFS shows is fragile under huge-page pressure. Suppose this baseline comes within run-to-run noise of our system on both ResNet-20 and HELR under a tight memory cap. That result would mean the layout and projection add nothing, and we would report the idea as refuted rather than reframe it.
Why not just use fewer keys, by composing rotations from a small base set? Baby-step giant-step decomposition, which KeyMemRT already applies, trades key count for extra key-switches. This is complementary to our work, not a replacement for it. Whatever key set remains still has to be stored and streamed. We will apply our layout on top of the reduced key set and report both effects.
Why not just compress the keys? Key material is close to uniformly random modulo each prime, so general-purpose compressors have little to exploit. Even on compressible data, Zstd costs up to 4.2x on NVMe relative to uncompressed scans (Columnar Storage Formats). We will include a Zstd run on the key file as a one-line check.
Proposed Experiments
The workloads are CKKS ResNet-20 and HELR, the two programs studied in simulation by Storage I/O Perspective, plus LeNet, AlexNet, and ResNet-50 from KeyMemRT, all at standard security parameters. They run on a GPU CKKS backend with keys on node-local NVMe and, separately, on a shared parallel file system.
The metrics are six quantities: end-to-end latency, time stalled waiting on keys, bytes read from storage, the number of open, stat, and close calls, metadata-server operations per second under concurrent tenants, and tail latency as the number of tenants grows at a fixed memory cap.
The first decisive check is whether storage matters on real hardware at all. If prefetching into a single file already hides all key I/O behind computation for these programs, the problem is simulated rather than real, and we stop.
The second check is the ablation pair. Level projection should reduce bytes read by roughly the fraction of key limbs belonging to already-dropped primes, which the compiler can compute in advance. The static plan should close most of the gap between per-limb files and a single file.
The third check is the multi-tenant scaling curve on the shared file system. Here the per-object designs should show metadata-server load growing with the tenant count, while ours should stay near one open per job.
References
- KeyMemRT Compiler and Runtime: Unlocking Memory-Scalable FHE
- The Future of Fully Homomorphic Encryption System: From a Storage I/O Perspective
- Characterization and identification of HPC applications at leadership computing facility
- Hvac: Removing I/O Bottleneck for Large-Scale Deep Learning Applications
- SplitFS: reducing software overhead in file systems for persistent memory
- An Empirical Evaluation of Columnar Storage Formats
- FLSSM: A Federated Learning Storage Security Model with Homomorphic Encryption
- BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery