Some on-device neural networks run partly inside a trusted execution environment (TEE). A TEE is a hardware-walled region of the processor, such as Arm TrustZone or Intel SGX, where code and data stay hidden even from the device's own operating system. Systems that do this, such as protecting a model from theft or from attacks that infer its training data, take an existing network like a VGG or ResNet and choose by hand where to cut it. Architecture search, the automated design of networks under hardware budgets, could instead design the network and the cut together. But search needs a cheap, accurate estimate of what each candidate costs to run. Inside a TEE that estimate is hard to get. Moving a tensor across the wall costs time roughly in proportion to its size, and exceeding the tiny secure memory (about 14MB usable on TrustZone, 90–128MB on legacy SGX) triggers encrypted paging and a sharp slowdown. Neither cost appears in any existing hardware lookup table.
We propose to build and test a cost model for split execution. It combines three things: per-operator timings measured inside the enclave, a measured price for crossing the boundary as a function of bytes, and a measured paging curve as a function of the secure-side working set. We will validate it against full on-device measurement over two standard search spaces. Then we use it to search jointly over architecture and split point. The central empirical question is whether the cheaper option is enough. That option is the "just sum the operator latencies" rule that works on bare-metal microcontrollers. Our claim is that it fails, and fails specifically near the memory cliff and when the split point moves. If it does not fail, we will report that, and it is itself a useful negative result.
Core Claim
Per-operator profiling has shown that, on flat-memory hardware, a block's latency is the sum of its parts. AtomNet measured thousands of fused Conv+BN+ReLU kernels on STM32 microcontrollers and confirmed this kernel additivity. HW-NAS-Bench relied on the same additive assumption for its FBNet space across six plaintext devices. Neither profiled a secure world. Our claim is that in split TEE execution the cost of a candidate architecture $a$ cut at layer $s$ is well approximated by
$$ T(a,s) = \sum_{i \le s} t^{R}_i + \sum_{i > s} \rho\, t^{R}_i + c_0 + c_1 B_s + P\!\left(W_s - M\right). $$
Here $t^R_i$ is the plaintext (normal-world) time of operator $i$. The factor $\rho$ is a per-operator-type slowdown measured once inside the enclave. The constants $c_0$ and $c_1$ are the fixed and per-byte prices of a world switch. $B_s$ is the size in bytes of the tensor crossing at the cut. $W_s$ is the peak secure-side working set, meaning the weights plus live activations after the cut. $M$ is the usable secure memory. $P$ is a measured paging-penalty curve that is zero for negative arguments.
Apart from a handful of microbenchmarks, every term can be computed from the architecture graph without running the candidate. The estimator therefore costs no more than a lookup table, yet captures the non-additive pieces. The novelty is not a TEE benchmark by itself. It is the evidence of exactly where additivity breaks in enclaves, and the resulting ability to search over split point and topology together. No prior system does the latter.
Relation to Prior Work
Four enclave systems motivate the gap, and all of them partition fixed networks by hand. DarkneTZ runs the last layers of AlexNet, VGG-7 or ResNet-50 in OP-TEE on TrustZone. Hiding only the final layer costs 3% CPU time and drops membership-inference precision from about 85% to about 50%. Filling the roughly 14MB of secure memory costs up to 10%. MirrorNet places a small companion network in the TEE. Its regression-based performance emulator reaches 0.48% error on VGG-16, which shows that profiling-based prediction can work in TrustZone for a fixed family. It also warns that last-layer hiding is fragile, because an attacker can retrain a replacement head with about 1% of the data. Goten splits linear and non-linear operations between an untrusted GPU and SGX, using operation fusion to reduce boundary traffic. Marshaled Learning partitions large models across several enclave clients by greedy configuration. All four take the topology as given. We make topology and cut point search variables, and we generalize MirrorNet's emulator from one channel-scaled family to search spaces with thousands of cells.
The search-cost literature supplies the method and the constraints. NAS-Bench-201 provides 15,625 fully trained cells; its best CIFAR-10 test accuracy is 94.37%. Because accuracy is free there, any gain we report comes from cost modelling rather than from training luck. The same paper constrains us. It documents the collapse of differentiable search into skip-connection-heavy cells, which is why we do not rely on a supernet with a penalty term. Rethinking Performance Estimation in NAS and EZNAS show that accuracy can be estimated cheaply: BPE achieves a 1,000x search speedup, and EZNAS reaches a Kendall rank correlation of 0.65 on NAS-Bench-201 CIFAR-10. We reuse these for the accuracy side of the FBNet space. On the profiling side, Zhang et al. found 50x latency swings across mobile runtimes caused by memory preparation and type conversion. That result is a warning that the in-enclave slowdown $\rho$ may differ by operator type, and is why we measure it per type rather than as one global factor.
Research Program
We build the cost model on two platforms. The first is a Raspberry Pi 3B running OP-TEE, the platform MirrorNet used, with roughly 16MB of secure memory. The second is an Intel SGX machine with a 90–128MB enclave page cache.
On each platform, a small microbenchmark suite measures four things. First, a world switch carrying tensors from a few bytes to tens of megabytes, which gives $c_0$ and $c_1$. Second, each operator type in the NAS-Bench-201 and FBNet vocabularies run on both sides, which gives $\rho$ per type. Third, a synthetic workload whose secure working set is swept across $M$, which gives $P$. Fourth, pairs and triples of adjacent operators inside the enclave, which check whether additivity holds there apart from paging.
Ground truth comes from full measurement. We sample about 2,000 (cell, split) pairs from NAS-Bench-201 at CIFAR-10 and ImageNet-16-120 resolution. We sample a further set of FBNet-space models at 224-pixel input, deliberately weighted toward working sets near $M$. NAS-Bench-201 models are small enough that they rarely touch the memory limit, so the FBNet space is where the paging term is tested.
Search uses regularized evolution over the joint space of architecture and split point. It minimizes estimated split latency under a security constraint. The default constraint is that a minimum fraction of parameters, including the classifier and at least one feature block, sits in the enclave, which goes beyond DarkneTZ's last-layer rule. On NAS-Bench-201, accuracy comes from the tabulated results. On FBNet, it comes from a BPE-style short training run, and the final Pareto candidates are trained fully. Security is measured only on finalists, because it is expensive. We report membership-inference precision following DarkneTZ, and accuracy recovered by an attacker retraining the hidden part from 1% of the data following MirrorNet.
Baselines
The naive reference point is FLOPs. Candidates are ranked by total FLOPs and the network is cut at the last layer. HW-NAS-Bench reports a FLOPs-to-latency Kendall correlation as low as 0.18 on the Edge TPU and below 0.5 on several devices in plaintext. We expect at least this poor a showing in the enclave, and the proposal must clearly beat it to matter at all.
The strongest existing cost estimator is a plaintext lookup table, the HW-NAS-Bench approach. We use the Raspberry Pi 4 latencies for NAS-Bench-201 and FBNet, with per-block sums for FBNet, rescaled by one global in-enclave factor fitted on the same microbenchmark budget we allow ourselves. This is the closest off-the-shelf competitor, because it is profiled, public and additive. For fairness it receives the same number of on-device measurements for fitting as our model. Separately, no reported result was found on Lacuna for latency-prediction accuracy of any method inside a TEE over a NAS search space. The nearest reported figure is MirrorNet's 0.48% emulation error on a single VGG-16. We will run MirrorNet-style per-family regression on our samples as a second competitor.
The strongest existing end-to-end method is "search in the clear, then split by hand". We run the same evolutionary search with the plaintext table to find the best plaintext architecture, then apply the DarkneTZ rule of hiding the last layers until secure memory is full. It uses the same search budget, the same number of evaluations, and the same accuracy source.
Two ablations isolate the key ingredient. The first is an in-enclave additive table: AtomNet-style kernel additivity with per-type $\rho$, but no boundary or paging terms. The second is the full model without the paging term.
We count the proposal as beating the baselines under three conditions. Kendall correlation over all sampled pairs must exceed the plaintext table's by a margin larger than its bootstrap interval. The top-decile Kendall correlation and the rate of constraint violations, meaning candidates predicted to fit but actually paging, must improve over the additive ablation. At matched measured latency, jointly searched models must reach higher accuracy, or lower attacker-recovered accuracy, than search-then-split.
Why not just…?
Why not just constrain FLOPs or parameter count? Plaintext evidence already rules this out as a latency estimator, with a FLOPs–latency correlation as low as 0.18 in HW-NAS-Bench. Parameter count is related to the memory term, but it ignores activation size, and activations drive both $B_s$ and the working set. The FLOPs baseline above measures exactly how badly it fails inside an enclave.
Why not just sum in-enclave operator latencies, as AtomNet does on microcontrollers? This is the real threat. For small NAS-Bench-201 cells that never approach $M$ and always cut at the same layer, it may be nearly as good as our model, because the boundary term is then almost constant. We expect it to break in two regimes. One is when the cut point varies, because $B_s$ then changes by orders of magnitude across layers. The other is when working sets approach secure memory. We will report the correlation gap separately inside and outside these regimes. If the gap is negligible in both, the conclusion is that additivity transfers to enclaves, and the contribution shrinks to a released in-enclave table. We would state that plainly.
Why not just search in plaintext and then split by hand, as DarkneTZ does? DarkneTZ's 3–10% overhead suggests this is cheap when the cut is late. However, MirrorNet shows that late cuts are weak against retraining attacks. Hand splitting also never considers architectures that place a narrow tensor exactly where an earlier, more protective cut would go. The search-then-split baseline tests whether joint search finds a better latency–protection trade-off. A tie would mean topology does not need to know about the enclave.
Why not just add a TEE latency penalty to a differentiable supernet? NAS-Bench-201 documents skip-connection collapse in this kind of search. The paging penalty is also a step function, which gradients handle poorly. Our discrete search with a closed-form cost model avoids both problems.
Decisive Checks
The first decisive check is the cost-model correlation table across the two platforms and two spaces. It is broken down by distance of the working set from $M$ and by cut depth. It shows where, if anywhere, additivity fails. The second is constraint violation in the FBNet space. If the additive ablation's predicted-feasible candidates page on hardware at a meaningful rate while ours do not, enclave-aware profiling is necessary. The third is the search outcome. Jointly searched NAS-Bench-201 models should be compared with search-then-split at equal measured enclave latency, using the tabulated accuracy, against the 94.37% best-cell ceiling on CIFAR-10. The finalists also get membership-inference and 1%-retraining attack scores. This tells us whether moving the boundary into the architecture buys protection that hand cutting cannot.
References
- AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- DarkneTZ: towards model privacy at the edge using trusted execution environments
- MirrorNet: A TEE-Friendly Framework for Secure On-Device DNN Inference
- Goten: GPU-Outsourcing Trusted Execution of Neural Network Training
- Marshaled Learning: Bridging Large Neural Networks with Memory-Constrained Trusted Execution Environments in Federated Learning
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Rethinking Performance Estimation in Neural Architecture Search
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- A Comprehensive Benchmark of Deep Learning Libraries on Mobile Devices