Slack-Aware Encrypted Inference for Real-Time Systems

Homomorphic encryption lets a server run a neural network on data it cannot read. The price is enormous: encrypted inference runs thousands of times slower than ordinary inference. Real-time systems, such as drone autopilots, robot controllers, and industrial edge boxes, have the opposite requirement. Every periodic job must finish before a fixed deadline, so they are provisioned for the worst case and then sit idle most of the time. This note proposes to put those idle cycles to work. The encrypted network is built from swappable modules, each available in several versions that differ in cost and accuracy. At every module boundary, the real-time scheduler picks the most accurate next version that still provably meets the deadline, using the time left over when co-located tasks finish early.

The idea works because of a property that ordinary code lacks. Encrypted computation is data-oblivious: it executes the same sequence of polynomial operations whatever the hidden input is. The cost of each module version can therefore be computed almost exactly from its operation counts, and those costs compose without the 5x gap between worst-case and typical runtime that OP-DFI reports for ordinary control code. Encrypted inference is the expensive but predictable task. The surrounding plaintext tasks are cheap but unpredictable, and they are the ones that leave slack behind. The proposal pairs the two.

Core Claim

A modular encrypted network can be re-planned at each module boundary under a guaranteed time budget, using exact per-module cost models. On realistic task sets, this should deliver substantially higher average accuracy than the best statically provisioned encrypted network, with zero deadline misses. It should also beat both plain opportunistic execution and early-exit networks.

The planner contains one structural insight that generic anytime inference lacks. In leveled CKKS (an encryption scheme for approximate real-number arithmetic, where each multiplication consumes one of a fixed number of "levels"), operations get cheaper as the ciphertext descends levels. If an early module takes a cheap low-degree version, it saves levels, and the planner can then drop the ciphertext to a lower level early. That makes every later module cheaper. Version choices are therefore coupled across modules. A greedy per-module choice is not optimal, and the planner has to reason over levels and time jointly.

Relation to Prior Work

Two lines of work motivate the gap from opposite sides.

The first line cuts encrypted-inference cost offline and ignores scheduling. HEMET shows that modular blocks like SqueezeNet's Fire module increase multiplicative depth (the longest chain of multiplications, which sets encryption parameter size). It runs a greedy block-wise search and fuses coefficients to reduce latency. SAFENet keeps exact ReLUs only on important channels and searches per-layer polynomial degrees, at a cost of about 7 GPU hours. XNET packs ciphertext slots tightly and accelerates the polynomial arithmetic on a GPU. All three output one fixed graph.

The second line spends real-time slack on security but never on cryptographic computation. OP-DFI compiles several versions of code blocks with different memory-check granularity and dispatches the most secure version that fits the remaining reservation. Contego runs monitors at lowest priority in a passive mode and switches them to a budgeted server when an alarm fires. We borrow OP-DFI's multi-versioning-under-reservation mechanism. We differ in two ways. Our versions trade accuracy rather than coverage. And our cost model is exact rather than estimated from input context, because the workload is data-oblivious.

XNET uses "real-time" to mean about 32 ms latency. It offers no deadline or worst-case analysis, a distinction that Zhu & Liu's review makes sharply. Tran et al. achieve genuine 500 Hz encrypted control. They get there only with additive-only Paillier encryption and 256-bit keys on a hard-coded FPGA, so they cannot evaluate nonlinear policies. Our target sits between these two: nonlinear encrypted networks with deadlines in the tens-of-milliseconds-to-seconds range, on commodity multicore edge hardware.

Two results act as constraints on the design. HEMET's depth trap means modular software structure must not leak into circuit depth. We therefore apply coefficient merging inside each module, and module boundaries cost no extra level. ScheduLeak shows that schedule-dependent behaviour is observable. We therefore have to check what version choices reveal.

Research Program

The first component is a module interface in the spirit of DeepDIVA and UCEE. Each encrypted module declares four things: its set of versions, the levels each version consumes, its exact operation counts (multiplications, rotations, rescales), and the accuracy contribution measured on validation data. Versions vary the polynomial degree (2 or 3), the fraction of channels given the higher degree, and whether to mod-switch afterward.

The second component is a cost model. It maps operation counts at level $\ell$ to time on the target platform. It is calibrated on that platform under memory-bus interference from co-runners, so that it is a bound rather than an average.

The third component is a shared-weight network trained so that every combination of module versions is usable. This replaces SAFENet-style per-configuration search with one training run.

The fourth component is the run-time planner. Let $B$ be the remaining guaranteed time budget, $\ell$ the remaining levels, and $k$ the next module. The planner picks

$$v_k^\* = \arg\max_{v} \; \Big[ a_k(v) + A_{k+1}\big(\ell - d(v),\, B - c_k(v,\ell)\big) \Big],$$

where $a_k(v)$ is version $v$'s accuracy contribution, $d(v)$ its level cost, $c_k(v,\ell)$ its time at level $\ell$, and $A_{k+1}$ a precomputed table of the best achievable accuracy-to-go. Because costs are exact and discrete, the table is small and the decision costs microseconds.

The fifth component is a schedulability analysis. It reserves budget for the cheapest complete path, so deadlines hold even with zero slack, and it treats upgrades as consumption of reclaimed slack only.

On security, the version choice depends on co-runner timing, never on ciphertext contents. Plaintext confidentiality is therefore unaffected. What can leak is that the version sequence reveals co-located task behaviour to whoever sees it. We measure this explicitly.

Baselines

The naive reference point is a single static encrypted network: the most accurate version combination whose worst-case cost fits the guaranteed budget, compiled once with HEMET-style coefficient merging. This is what a careful engineer would deploy today. Beating it means higher mean accuracy at equal deadline guarantees.

The cheapest-everywhere network is the XNET regime: degree-2 activations throughout. On CIFAR-10 it reports 61.09% accuracy, with latencies of 31.862 ms on an A100 and 0.46 s on a single CPU thread (XNET). We re-run this configuration on our platform rather than comparing against those numbers.

The strongest offline method is a SAFENet-style channel-wise mixed-degree network searched for the guaranteed budget. SAFENet reports 85.1% on CIFAR-10 CNN-7, but in a hybrid encryption/secure-computation setting with a 61% latency reduction over Delphi (SAFENet). That figure is not directly comparable to pure CKKS, so we re-implement its allocation in our CKKS setting with the same training budget as our shared-weight network.

The opportunistic baseline is a Contego-style server (Contego). It runs the most accurate fixed network at lowest priority and falls back to the cheap network when a job would miss its deadline.

Two ablations isolate the key ingredients. The first is a greedy planner that ignores the level-to-cost coupling. The second is a planner that uses averaged rather than calibrated costs, which tests whether data-obliviousness actually buys tight bounds.

All methods share the same data, platform, task sets and training compute. For deadline-miss rate of encrypted inference under real-time scheduling, no reported result was found on Lacuna. Every number in that comparison will be our own.

Why not just…?

Why not just provision statically for the best network that fits? This is the main threat. It wins whenever co-runners rarely finish early. OP-DFI's observation that worst-case time can be 5x the typical runtime suggests that slack is usually plentiful. The experiment sweeps co-runner execution-time distributions from tight to loose and reports where the dynamic gain vanishes. If the gain is small on realistic TACLeBench-derived task sets, the claim fails.

Why not just run on a GPU fast enough that scheduling doesn't matter? XNET's 31.862 ms comes from an A100 with a crude activation and 61.09% CIFAR-10 accuracy. Edge platforms lack such hardware, and XNET's own CPU figure is 0.46 s. Faster hardware shifts the operating point but does not remove the accuracy/cost tension. Our method is orthogonal to it.

Why not just use an early-exit network? This is the closest alternative, and we treat it as a real competitor. Early exits produce a coarse answer and then refine it. Under encryption, however, a skipped tail still forces depth to be fixed at encryption time, and the client must decrypt and choose between outputs. Early exits also ignore the level coupling that our planner exploits. We implement an encrypted early-exit variant at matched depth. If it matches our accuracy–budget curve, the per-module version space is unnecessary and we would say so.

Why not just use a trusted enclave? MiniTEE shows that standard TrustZone world switches hurt determinism. Enclaves also require trusting the silicon. They are the right answer when that trust is acceptable, and we do not claim otherwise.

Proposed Experiments

The decisive first check is whether data-obliviousness actually yields tight bounds. We measure the execution-time distribution of each CKKS module version, alone and under memory-heavy co-runners, on a multicore ARM edge board and an x86 box. If the spread under interference approaches that of ordinary code, the exact-cost premise fails, and the method reduces to OP-DFI with estimated slack.

The main experiment trains networks on MNIST, Fashion-MNIST and CIFAR-10, the benchmarks used by XNET, HEMET and SAFENet. These networks run as periodic encrypted tasks alongside two kinds of co-runners: UUniFast-generated task sets as in Contego, and TACLeBench programs to supply realistic run-time variance. Periods range from 100 ms to 2 s, and utilization is swept. Metrics are deadline misses (required to be zero), mean and 5th-percentile accuracy over jobs, and the fraction of jobs that are upgraded.

A schedule trace in the spirit of the Ardupilot evaluations then runs a perception task at fixed rate beside autopilot-like control tasks. This confirms that the control tasks' response times do not change.

Finally, an attacker in the ScheduLeak style observes either the version sequence or the encrypted task's preemptions. We measure how well that attacker infers co-runner arrival times compared with the static baseline. If version choices leak substantially more, we test quantizing the decision points as a mitigation and report its accuracy cost.