Phones and cars increasingly need models that keep learning from a user's data after deployment. The standard defense against forgetting is a replay buffer, a small store of past examples that the model rehearses alongside new data. The same setting also calls for a trusted execution environment (TEE): a hardware-isolated region of memory and compute that the device's ordinary operating system cannot read or tamper with. On Arm TrustZone that region is tiny, about 14MB usable for applications according to DarkneTZ. So the buffer has to sit mostly outside the TEE. It is then pulled across the boundary at every training step, where it pays copying, decryption, and integrity-checking costs. Worse, the pattern of which records get read reveals which examples the learner considers hard or worth keeping.
We propose to separate two decisions that current replay methods tie together: what to keep in the buffer and when to read it. What to keep stays data-dependent and is decided inside the TEE from cheap forward-pass scores. When to read becomes a fixed pseudorandom schedule, committed ahead of time and independent of the data. A fixed schedule lets the TEE prefetch and decrypt upcoming replay batches while the current step computes, which hides boundary latency. The same schedule means the untrusted operating system sees the same access trace whatever the buffer contains. The claim is that one design choice buys both speed and access-pattern privacy, at an accuracy cost we expect to be small and will measure.
Core Claim
In a partitioned model with a frozen feature extractor in the ordinary operating system (the rich execution environment, REE) and the trainable classifier head inside TrustZone, continual learning can keep its replay buffer as fixed-size, authenticated-encrypted records in REE memory. Three properties then hold together. First, replay adds almost no stall time, because a data-independent read schedule allows pipelined fetch-and-decrypt. Second, the REE learns nothing about buffer contents, retention decisions, or labels from memory traffic, and cannot tamper with or roll back stored examples without detection. Third, accuracy stays within a small margin of a data-dependent coreset method such as CSReL that runs without any isolation. Pipelining hides the fetch whenever the per-batch fetch plus decrypt time $t_f$ is no larger than the per-step head compute time $t_c$. We will measure both on hardware and report the batch sizes at which $t_f \le t_c$ holds.
What Is Different From Prior Work
The pieces exist separately; the combination does not. Several papers supply the method. TZ-LLM hides TrustZone memory allocation and decryption latency by prefetching model weights along the fixed layer order of LLM inference. It explicitly leaves data-dependent access as future work. Our move is to make replay access fixed by construction, rather than to predict a random access pattern. Continual Learning with Foundation Models: An Empirical Study of Latent Replay shows that storing frozen-encoder features instead of images cuts compute by about 100x. That is what makes records small enough to stream, but the paper assumes ordinary RAM with no boundary. CSReL's reducible loss is the current model's loss minus the loss of a "holdout" model trained on the current task. It needs only forward passes and resists noisy labels. Its drawback is the holdout model, and in latent space that model is just a linear head, which is cheap enough to train inside the TEE.
Other papers motivate the problem. DarkneTZ puts the final layers of a network in TrustZone and shows this drops membership-inference precision from about 85% to about 50%. It works on static single-task data, with no replay. Marshaled Learning splits large models across memory-limited TEE clients in federated learning. It has no continual stream and no buffer. Bindox and DUUI show that serialization and copying across a boundary can dominate the actual work. Bindox maps buffers to shared physical pages, but it trusts a kernel driver and does not cross a hardware trust boundary.
The side-channel literature constrains the design. SgxPectre and Characterizing Trust Boundary Vulnerabilities in TEE Containers show that memory access patterns leak enclave state. That rules out any design that reads replay records in a data-dependent order, which includes every prioritized-replay variant.
Threat Model and Setting
The feature extractor runs in the REE, so an attacker who controls the REE sees each input's features at the moment they are computed. We do not claim to hide that. We protect four other things. The first is the labels and the head's parameters, following DarkneTZ. The second is the retained history, including which samples were kept, their labels, and their weights, so that a later REE compromise or a storage dump cannot recover it. The third is the integrity and freshness of the buffer: the REE must not be able to poison records or roll them back undetected. The fourth is the access trace, which must not reveal retention or hardness.
Each record is a fixed-size authenticated-encrypted block containing a quantized latent, a label, a weight, and a version counter. The TEE keeps the key and a small table of counters. Every slot is rewritten on a fixed rotation, carrying either a real replacement or a re-encryption of the old contents. As a result, evictions look the same as non-evictions to an observer.
Research Program
We will build a prototype on an OP-TEE-capable Arm board. The trusted component contains four parts: the linear or small MLP classifier head; a holdout head for reducible-loss scoring on the current task's recent latents; the key and counter table; and a pipeline that decrypts batch $k+1$ from OP-TEE shared memory while batch $k$ trains.
Admission and eviction decisions are made inside the TEE from reducible loss. Only their effect, a re-encrypted slot, leaves the TEE. The replay schedule is a keyed pseudorandom permutation over slots, drawn one epoch ahead. Record size is fixed, and we treat it as a design variable. For a given boundary bandwidth, is it better to keep more slots with 4-bit latents or fewer slots with 8-bit latents?
The analysis has three parts. The first asks how much accuracy is lost by forbidding data-dependent replay sampling, given that data-dependent admission is still allowed. The second measures stall fraction across batch sizes and encoder widths, which tells us where the pipelining condition fails. The third runs a trace-based adversary against both our design and a priority-sampling design.
Baselines
The naive reference point keeps a reservoir-sampled latent buffer as plaintext in REE memory. Batches are copied into the TEE through ordinary shared-memory calls, with no encryption and no pipelining. This is the fastest insecure option and sets the lower bound on overhead. It runs on the same board with the same encoder, buffer size, and learning-rate sweep.
The strongest accuracy competitor is CSReL running entirely in ordinary memory with no TEE. The file reports its final average accuracy as 39.82% on Split CIFAR-10 and 18.47% on Split CIFAR-100, with greedy coreset selection at 37.68% and 15.04%. Those are image-level numbers. We will rerun CSReL and a GCR-style gradient-matching coreset (GCR) in our latent setting with the same buffer size and tuning budget, rather than compare against the published values. Matching rerun CSReL to within about one accuracy point, while adding isolation, counts as success.
The strongest systems competitor naively adapts TZ-LLM-style prefetching to prioritized replay. It prefetches the currently highest-priority slots and falls back to on-demand fetches when the prediction misses. I found no reported result on Lacuna for replay-buffer streaming across TrustZone. That is why this baseline must be built, and our design beats it if it has lower stall time and trace leakage near chance.
Three ablations isolate the key ingredient. The first keeps our encryption but replaces the fixed schedule with data-dependent sampling. The second keeps the schedule but removes pipelining. The third keeps everything but swaps reducible-loss admission for reservoir or highest-loss admission. The first ablation measures what the fixed schedule costs in accuracy. The second measures what pipelining buys in speed. The third measures whether the selection score matters.
Why not just…?
Why not just store quantized latents entirely inside the TEE? This is the most serious threat, and we treat it as a real competitor rather than a straw man. A 768-dimensional 8-bit latent is under 1KB, so several thousand might fit in 14MB alongside a linear head. We will sweep buffer size and encoder width to locate the crossover. That crossover moves if the head grows to an MLP, if the cut point moves into the encoder, or if several applications share the secure world. Moving the cut deeper is exactly the remedy the latent-replay study's roughly 20% out-of-distribution gap points toward, and it makes stored features larger. If the in-TEE buffer covers realistic regimes at equal accuracy, we will report that plainly. The contribution then narrows to the large-buffer and deep-cut regime.
Why not just use OP-TEE shared memory, which is already zero-copy? We do use it as the transport. Zero-copy removes duplication, but it neither hides contents nor protects integrity, and it does nothing about the access trace. The second ablation separates what shared memory provides from what the schedule and encryption add.
Why not run everything in a confidential VM? Marshaled Learning reports 1-3x overhead on Azure confidential VMs and notes the much larger trusted code base. Edge TrustZone devices also cannot provision such VMs. We therefore do not run this baseline. We only note it as the cloud alternative.
Why not add differential privacy instead? Differential privacy protects the model's outputs, not the integrity of the stored history or the access trace. DarkneTZ also argues it costs substantial accuracy. We include a differentially private SGD head as a secondary comparison for membership-inference precision only.
Why not just use reservoir sampling for admission? Reservoir sampling is a strong baseline in continual learning. It may match reducible loss on clean streams. The third ablation will show this, and the label-noise test is where CSReL reports that competitors collapse.
Proposed Experiments
The streams are Split CIFAR-10 and Split CIFAR-100, plus the in-distribution and out-of-distribution CIFAR-100 and Aircraft streams from the latent-replay study. We add a CIFAR-100 variant with 20% corrupted labels following CSReL. Metrics are final average accuracy, forgetting, per-step wall time, stall fraction, peak secure memory, and bytes crossing the boundary per step. We also run membership-inference precision against the head, using DarkneTZ's attack, where roughly 50% indicates success.
The decisive checks are three. First, a trace adversary, a classifier given the REE-visible read and write sequence, should predict whether a given sample was retained no better than chance under our schedule. It should do clearly better under data-dependent sampling. Second, stall fraction should stay near zero wherever $t_f \le t_c$, and should degrade exactly where the measurement says pipelining fails. Third, the accuracy cost of the fixed schedule relative to data-dependent sampling should be within one point on the clean and noisy streams. If the third check fails, the central trade-off is worse than claimed, and the paper becomes a quantified negative result on oblivious replay.
References
- DarkneTZ: towards model privacy at the edge using trusted execution environments
- Coreset Selection via Reducible Loss in Continual Learning
- TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone
- Continual Learning with Foundation Models: An Empirical Study of Latent Replay
- Marshaled Learning: Bridging Large Neural Networks with Memory-Constrained Trusted Execution Environments in Federated Learning
- Bindox: An Efficient and Secure Cross-System IPC Mechanism for Multi-Platform Containers
- Unlocking the Heterogeneous Landscape of Big Data NLP with DUUI
- SgxPectre Attacks: Leaking Enclave Secrets via Speculative Execution
- Characterizing Trust Boundary Vulnerabilities in TEE Containers
- GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning