When Latency Stops Predicting Energy: Measuring and Minimizing the Energy Cost of Confidential Inference

Running a neural network inside a trusted execution environment (TEE), a hardware-isolated region of a processor that even the operating system cannot read, is now the main practical way to protect proprietary model weights and private inputs on machines the model owner does not control. Every system built to make this practical judges itself by speed alone: time to first token, tokens per second, CPU seconds. The inference-energy literature, for its part, has measured joules carefully, but only for ordinary unprotected serving. In that setting latency and energy are almost the same quantity, with a reported correlation of $r = 0.996$ (Towards Sustainable NLP). This proposal tests whether that equivalence survives once a protection boundary is added. If it does not, the configurations the TEE literature picks as "fastest" may not be the ones that use the least energy.

The reason to doubt the equivalence is mechanical. A protected inference spends much of its time not computing. It switches between the normal and secure worlds, copies activations across the boundary, and encrypts and authenticates memory traffic. One system reports that these boundary crossings take over 60% of latency in naive split pipelines (AegisGuard). During such stalls an accelerator may sit near idle power, or a CPU may spin at full power in a polling loop. Either way, power is no longer roughly constant across the run, so wall-clock time no longer tells you how many joules were spent. We propose to measure the energy cost of confidentiality phase by phase on three TEE platforms. We then test whether choosing which layers to protect by energy, rather than by latency, changes the choice and how much energy that saves at a fixed security level.

Core Claim

Write the energy of one request as $E = \sum_k P_k t_k$. Here $k$ ranges over execution phases: untrusted accelerator compute, enclave compute, world switches, boundary copies with encryption, and stalls. $P_k$ is the average whole-system power in phase $k$ and $t_k$ is its duration. Latency is a faithful proxy for energy only when the $P_k$ are roughly equal, which holds approximately for monolithic GPU serving.

Our hypothesis is that in partitioned confidential inference the stall and transfer phases have systematically different power from compute. They may be lower, when the GPU idles while waiting on a slow enclave. They may be higher per unit of useful work, when CPUs busy-wait while several devices are held awake. The hypothesis makes three falsifiable predictions. First, within a family of protected configurations of the same model, the ranking by latency and the ranking by measured joules will disagree measurably. Second, the configuration that is optimal for latency under a fixed security constraint will not be the one that is optimal for energy. Third, the gap between them depends on workload geometry, meaning the ratio of input to output tokens. Autoregressive decoding crosses the boundary once per generated token, so the energy penalty of protection should grow faster with output length than its latency penalty.

If all three predictions fail, the useful result is a clean negative: latency is a safe proxy even under TEEs, and the existing literature's speed numbers can be converted to energy directly. We treat that outcome as a real possibility, not a strawman.

Relation to Prior Work

Four TEE inference systems motivate the work, and we reuse their configurations as test subjects. DarkneTZ splits CNNs between the Linux side and an Arm TrustZone enclave and reports 3–10% CPU-time overhead. It is the only paper in the file that measures power, and it does so only in aggregate, without attributing energy to world switches or memory protection. AegisGuard uses a learning agent to choose which LoRA adapters to protect inside the TEE while a GPU runs the frozen base model, and it reports speedups. SLIM avoids per-layer crossings by disguising activations with orthogonal transforms that survive normalization layers. TEE-X takes the opposite route and compresses an entire vision transformer to run inside a TrustZone enclave with SIMD vectorization. None of the four measures joules per request.

We do not propose a new protection scheme. We take their placements as the design space and ask whether energy reorders it. Memory-Efficient Deep Learning Inference in TEEs cuts secure page evictions from 1.8 million to about 8,270 by slicing feature maps. It also reports latency only, which leaves open whether the extra slicing work costs more energy than the paging it avoids.

Three papers constrain the design. SLIM states that its transforms break under INT4/INT8 quantization, so SLIM configurations must run in high precision. That forfeits the 25–55% power savings of quantization reported by ML-EcoLyzer, and the comparison has to account for it. NVIDIA GPU Confidential Computing Demystified shows that host-to-GPU transfers in confidential mode carry a timing side channel that scales with size. Any energy-saving change to transfer behavior must therefore be checked for whether it widens that channel. The SoK on ML with Confidential Computing leaves energy out of its taxonomy entirely, which confirms the gap but supplies no method.

The measurement method comes from the energy literature. From ML-EcoLyzer we adopt sampling at 5 Hz or faster, since it finds 1 Hz sampling overestimates emissions by nearly 6% on short inferences. From Energy Considerations of LLM Inference we adopt the prefill/decode split and trace-driven workloads. We also extend that paper's methodology, because it measures only GPU board power and explicitly excludes host CPU and memory. Under TEEs, those excluded components are exactly where the protection work happens. Beyond Test-Time Compute Strategies supplies the energy-per-token framing and notes that its energy curves are tied to specific hardware. It names confidential enclaves as an untested setting, which is the setting we test.

Research Program

The first component is a measurement harness that attributes whole-system energy to execution phases. It combines an external wall-socket power meter with on-chip counters: Intel's processor energy counters for CPU package and DRAM, NVIDIA's management library for the GPU, and the Jetson's onboard rail monitors for the edge platform. The harness timestamps phase boundaries by instrumenting the enclave entry and exit calls, the TrustZone secure-monitor calls, and the copy and encrypt routines, then integrates power over each phase. The harness first runs every model unprotected, so that each protected configuration has a matching native baseline and the extra energy can be stated as the energy cost of confidentiality.

The second component is a characterization across three platforms, chosen to cover the deployment styles in the file. The first is an Intel SGX host with a discrete GPU, running the AegisGuard and SLIM placements on OPT-2.7B, LLaMA-7B, Gemma3-1B, and LLaMA3.2-3B. The second is an NVIDIA Jetson AGX Xavier with OP-TEE, running TEE-X-style whole-model enclave execution of DeiT-Small next to a DarkneTZ-style split of the same model. The third is a Hopper GPU in confidential-computing mode, if we can get access, so the full model runs in the protected GPU region and the cost comes from encrypted bounce buffers rather than partitioning. On each platform we sweep input and output lengths independently and replay length distributions from the BurstGPT and Azure traces. Together these give energy-per-token curves under protection.

The third component turns the characterization into decisions. We fit a per-platform cost model that predicts joules for any layer placement from layer sizes, crossing counts, and measured phase powers. We then plug it into AegisGuard's selection procedure in place of its latency objective. The selection problem becomes: choose the set of shielded components that minimizes predicted energy, subject to surrogate-stealing accuracy staying at the level AegisGuard reports for full black-box protection. Alongside placement we expose one system knob, the stall policy (busy-wait polling versus blocking sleep at boundary crossings). It is the simplest lever that changes stall power without changing what is protected.

Baselines

The naive reference point is unprotected inference of each model with the same serving stack: vLLM where applicable, the same precision, and the same batch size. Every energy cost of confidentiality is stated relative to it. A second naive reference is the unoptimized TEE implementation each paper starts from. These are AegisGuard's "shield all adapters" setting, SLIM's comparison point ArrowCloak, TEE-X's naive enclave port, and the unmodified SGX VGG-16 run. Including them shows whether the published speedups translate into equal energy savings.

The strongest existing methods are the latency-optimized placements themselves, reproduced on our hardware.

AegisGuard is the strongest competitor for split LLM inference because it already optimizes exactly the placement question we ask, using latency as its objective. It reports a 3.33× speedup on LLaMA-7B over shielding all adapters and a cut in TEE memory from 224.4 MB to 54.96 MB. No reported result found on Lacuna for its energy per request.

SLIM is the strongest competitor for minimizing boundary crossings. It reports up to 13.80× faster time to first token and up to 4.23× higher throughput than ArrowCloak. No reported result found on Lacuna for its energy.

TEE-X is the strongest whole-model-in-enclave design. It reports DeiT-Small at 1.08 s in the enclave versus 1.02 s on the native GPU and 4.43 s for the naive port. No reported result found on Lacuna for its energy on the Jetson.

All of these baselines run on the same machines, the same datasets, and the same request traces, with the harness attached and identical warm-up and repetition counts. Our energy-objective planner gets the same search budget that AegisGuard's selection procedure uses.

The key ablation runs our planner with its cost model swapped back to measured latency, keeping the search, the security constraint, and the candidate set unchanged. Any difference between the two outputs isolates the effect of optimizing for energy rather than time. A second ablation holds the AegisGuard placement fixed and changes only the stall policy, to separate placement effects from waiting-behavior effects.

Beating the strongest baseline means the following. At equal surrogate-stealing accuracy (within the ~39% level AegisGuard reports) and with task accuracy within 0.5 points, the energy-objective placement uses at least 15% fewer joules per request than the AegisGuard placement on at least one model and trace. Its latency must also stay within 10% of AegisGuard's. Smaller differences would mean latency is an adequate proxy in practice.

Why not just…?

Why not just use latency, since latency and energy correlate at $r = 0.996$? This is the main threat, and the study is designed to settle it rather than argue against it. That correlation comes from native GPU serving (Towards Sustainable NLP), where every phase is compute on one device. We will report the same correlation, and the rank agreement between latency and energy, within each protected configuration family. If agreement stays near 0.99 and the latency-objective ablation picks the same placements, we publish that latency is a valid proxy under TEEs and drop the planner.

Why not just attach CodeCarbon or ML-EcoLyzer to the existing systems? Those tools mainly read accelerator power. Energy Considerations of LLM Inference explicitly excludes host CPU and memory, which is where enclave computation, memory encryption, and world switches draw power. We will run GPU-only telemetry next to the wall meter on every configuration and report how much of the confidentiality cost it misses. If it misses little, the simpler tools suffice and we will say so.

Why not just switch boundary waits from polling to sleeping and skip placement optimization? This could plausibly capture most of the savings, and the stall-policy ablation tests it directly. If changing the stall policy alone matches the energy-objective planner, the contribution reduces to a systems recommendation and a measured explanation of why it works. That is still useful, but a narrower claim. Changing stall behavior also changes the power and timing profile an untrusted host can observe. Given the timing channel found in NVIDIA GPU Confidential Computing Demystified, we will check whether host-visible power traces under each policy reveal output length better than chance.

Why not just run everything on a confidential GPU and avoid partitioning? On platforms that support it, this may well be the energy winner. That is why it is a configuration in the study rather than a dismissed alternative. Its remaining cost comes from encrypted bounce buffers whose energy is unmeasured. It is also unavailable on the edge devices where TrustZone splits are used, so it cannot replace the placement question there.

Decisive Experiments

The first decisive check is rank disagreement. For each model and platform we enumerate 20–50 security-equivalent placements and compute Kendall's rank correlation between latency and measured energy. Values well below 0.9 support the core claim, and values near 1 refute it.

The second check is the planner comparison on LLaMA-7B and OPT-2.7B fine-tuned on CommonSenseQA, replaying BurstGPT length distributions. We report joules per request, joules per output token, latency, task accuracy, and surrogate-stealing accuracy for the AegisGuard placement, the latency-objective ablation, and the energy-objective planner.

The third check is workload geometry. We fix input length and grow output length from 16 to 1,024 tokens for SLIM-style and AegisGuard-style runs, and test whether the energy cost of confidentiality grows faster than the latency cost, as per-token boundary crossings would predict.

The fourth check is the whole-model versus split comparison on the Jetson. We measure TEE-X-style enclave DeiT-Small against a split DarkneTZ-style placement on CIFAR-10 and ImageNet. The question is whether the joule ordering matches the latency ordering, given TEE-X's 1.08 s against the native 1.02 s.

The fifth check revisits the SGX VGG-16 slicing result. We measure whether reducing evictions 200-fold reduces energy by a comparable factor, or whether the overhead of slicing and stitching eats part of the gain.

Together these settle whether confidential inference needs its own energy accounting, or whether its existing speed results already answer the energy question.