Federated learning lets many devices train a shared model by sending gradient updates instead of raw data. To save bandwidth, devices often send only their few largest gradient entries, a practice called sparsification. A common way to keep the server from seeing any single device's update is to aggregate inside a trusted execution environment, or TEE: a hardware-isolated region of the processor, such as Intel SGX, whose memory the host cannot read. The host still shares caches and page tables with the enclave, though, so it can watch which memory addresses the enclave touches. With sparse updates, those addresses are the coordinates each device chose to send, and that choice depends on the device's labels. OLIVE shows that a server watching these access patterns infers labels with over 90% accuracy on CIFAR-100 at 1.25% sparsification. Its fix makes the enclave's memory trace fully independent of the data, which costs oblivious sorting over the whole model every round.
This note proposes a cheaper, accounted alternative. The enclave would deliberately leak something, but only an aggregate quantity: a noisy histogram of how often each model coordinate was updated across all clients in a round. That leak is then charged to the same differential-privacy budget that already covers the gradient values. Differential privacy (DP) is a guarantee that the output changes little when one person's data changes, measured by a budget $\epsilon$. Under this design the side channel stops being an unbounded hole. It becomes one more noisy release, and its cost should shrink as models grow, because it no longer scales with the full parameter count.
Core Claim
A TEE aggregator for sparse federated updates can replace full obliviousness with a "shuffle, then noisy-touch" design, which we call differential obliviousness: the memory trace may depend on the data, but only in a way that satisfies DP. The aggregator first obliviously shuffles all (coordinate, value) pairs received in a round, which breaks the link between a client and its coordinates. It then updates model coordinates directly, interleaving dummy accesses drawn from a calibrated noise distribution. The host learns only a DP-protected histogram of coordinate counts.
Writing $\rho_{\text{val}}(\alpha)$ for the Rényi-DP cost per round of the noised gradient values released to the model, and $\rho_{\text{idx}}(\alpha)$ for the cost of the noisy coordinate histogram exposed through the memory trace, the total cost over $T$ rounds at Rényi order $\alpha$ is
$$\rho_{\text{total}}(\alpha) = T\,\big(\rho_{\text{val}}(\alpha) + \rho_{\text{idx}}(\alpha)\big).$$
This follows from standard Rényi composition, since each round makes two releases. The claim has three parts. At a fixed total budget, this design matches fully oblivious aggregation in accuracy and in measured attack resistance. Its aggregation time grows roughly as $O(nk\log(nk))$ in the number of clients $n$ and the per-client sparsity $k$, plus the dummy accesses, rather than with the model dimension $d$. And the speedup over full obliviousness therefore widens as $d/(nk)$ grows, which is exactly the regime of large Transformer models.
How This Sits Against Prior Work
The motivation comes from two attack results. OLIVE (also Olive) establishes the label-inference side channel in sparse aggregation. It answers with constant-trace aggregation built on conditional-move instructions and bitonic sorting, keeping full utility and running roughly 10× faster than PathORAM, a general-purpose oblivious memory scheme. It deliberately avoids DP noise. Our design inverts that choice: we accept a bounded, accounted leak in exchange for removing the dependence on $d$.
SNPeek shows on AMD SEV-SNP confidential virtual machines that adding DP noise to an application's outputs does not stop trace-based distinguishing attacks. It also shows that constant-time oblivious memory implementations leak through ciphertext patterns. This result constrains the proposal rather than contradicting it. Noise on outputs fails because the trace is a separate channel that nobody accounted for. We put noise on the trace itself. We must also be explicit that our guarantee covers only the leakage we model, which is the coordinate-level access pattern.
The method comes from Doquet. It builds differentially oblivious select and join operators for database queries inside SGX, gives $(\epsilon,\delta)$ bounds over access patterns, and reports up to an order-of-magnitude speedup over fully oblivious memory. Doquet handles static relational data with no learning and no composition over many rounds. We carry the idea to gradient aggregation repeated over hundreds of rounds, and we fold the access-pattern cost into the training privacy accountant.
On the sparsification side, FL-TOP shows why sparsity matters for privacy and utility: perturbing only $K$ selected weights raises the signal-to-noise ratio under DP. It reports 0.81 accuracy versus 0.56 for standard DP federated learning on Fashion-MNIST at a 0.5% ratio. Its coordinates are chosen once from public data, so they leak nothing. That makes it a serious competitor, discussed below.
Homomorphic encryption, which computes directly on encrypted data, is the main cryptographic alternative to a TEE. EdgeDetect aggregates binarized gradients under Paillier encryption and lowers gradient-inversion PSNR from 31.7 dB to 15.1 dB. Zhou et al. show that even with clipping removed, DP training under fully homomorphic encryption is limited to convex models. Their trick cuts multiplicative depth per iteration from 24 to 9. Homomorphic aggregation hides coordinates only if updates are dense, because sending sparse indices in the clear reintroduces the leak. We therefore treat it as the "pay in compute and bandwidth instead of budget" point on the frontier. We measure it but do not build on it.
Research Program
The first component is the aggregator itself, implemented in SGX. Each round, the enclave receives $n$ client messages of $k$ (coordinate, value) pairs each. It performs an oblivious shuffle of the $nk$ pairs, reusing Olive's sorting primitives but over $nk$ items instead of $nk+d$. It then adds Gaussian noise inside the enclave to the summed values for the central DP guarantee.
For the trace, the enclave samples a dummy count for each coordinate bucket from a discrete Gaussian distribution. It then performs real and dummy updates in a random order, so the observable histogram is the true count plus noise. Coordinates are grouped into buckets at the granularity the attacker can actually resolve, such as a page or a cache line. That way the noise is spent on what is observable and not on finer detail the host cannot see. Sensitivity is bounded by enforcing exactly $k$ entries per client, so one client changes the histogram by at most $k$ unit counts.
The second component is the accountant. It tracks $\rho_{\text{val}}$ and $\rho_{\text{idx}}$ jointly and lets the operator split a fixed total $\epsilon$ between gradient values and the trace. We will study that split empirically, because it is the new tuning decision this design introduces.
The third component is the attack harness. We will reimplement Olive's label-inference attack on page-level traces, which are cheap for the host to obtain, and on cache-line traces collected with Prime+Probe. Prime+Probe is a technique where the attacker fills a cache set and times its own reloads to detect the victim's accesses. We will also add a learned trace classifier in the style of SNPeek's LSTM analysis, so that our evaluation does not rest only on the attack we designed the defense against.
Baselines
The naive reference is non-oblivious sparse aggregation inside SGX, with central DP noise on gradient values only. It is the fastest configuration and defines the attack ceiling. Olive's reported figure for this setting is over 90% label-inference accuracy on CIFAR-100 at 1.25% sparsification, and our reimplementation must reproduce roughly this before any defense comparison counts.
The strongest existing method is OLIVE, the fully oblivious aggregator. The file reports it keeps full baseline utility and runs about 10× faster than PathORAM. It does not report absolute aggregation times at Transformer scale; no reported result found on Lacuna for Olive's latency beyond CIFAR-scale models. It is the right competitor because it already solves the threat with no privacy cost.
We will run it on the same enclave, client count, sparsity, and model, with the same DP noise on values. We beat it if two things hold together. Aggregation time must be at least 3× lower once $d/(nk)$ exceeds about 100. And at an equal total $\epsilon$, test accuracy must be within one point while attack accuracy stays within one point of chance plus the advantage the $\epsilon$ bound permits.
The fixed-public-mask baseline is FL-TOP. Its coordinates leak nothing, so it needs no obliviousness at all. We give it the same public proxy-data budget and the same total $\epsilon$. In our design, that entire budget goes to the values.
The cryptographic baseline is dense Paillier aggregation in the EdgeDetect style. We also run a CKKS variant; CKKS is a homomorphic scheme for approximate arithmetic on real numbers. We report client compute, bandwidth, and server time at matched accuracy. No reported result found on Lacuna for homomorphic aggregation latency at these model sizes, so we measure it.
The key ablation keeps the oblivious shuffle but removes the dummy accesses. This isolates whether the noisy histogram is necessary, or whether breaking the client-to-coordinate link alone already defeats the attacks we test. A second ablation keeps the dummy accesses and removes the shuffle, which turns the trace leak into a per-client one.
Why not just…?
Why not just use Olive, since it is already 10× faster than PathORAM? This is the main threat. If Olive's overhead at ViT or small-language-model scale is already within about 1.5× of non-oblivious aggregation, the argument for spending privacy budget on the trace collapses. In that case we would report a negative result. The scaling experiment below settles it directly by sweeping $d$ while holding $nk$ fixed. Our prediction depends on Olive's sort touching all $d$ coordinates, and that is what the sweep tests.
Why not just use a fixed coordinate mask chosen on public data, as FL-TOP does? Then the trace leaks nothing, for free. The cost is adaptivity. Each client's most informative coordinates differ, especially when data are non-identically distributed, and a static mask cannot follow them. We will compare accuracy at matched $\epsilon$ and matched bandwidth under label-skewed client splits. If the fixed mask is within a point everywhere, adaptive sparsity is not worth defending, and we will say so.
Why not just encrypt the updates homomorphically and skip the TEE? Sparse homomorphic aggregation still reveals indices unless updates are padded to dense. Dense encrypted aggregation removes the bandwidth savings that motivated sparsity. Encrypted training beyond convex models also remains open per Zhou et al.. LIA-SA further shows that a server which tampers with Batch Normalization parameters recovers labels with 100% accuracy through secure aggregation. Encryption alone therefore does not close active attacks either. Our design does not close them either, and we state this as a shared limitation rather than an advantage. The matched-accuracy cost measurement places homomorphic encryption on the same frontier.
Why not just make clients choose their coordinates with a local DP mechanism, so the trace is harmless post-processing? Local DP on a $k$-of-$d$ choice requires far more noise than central DP on an aggregate histogram, because each client's selection is protected on its own. We include noisy local top-$k$ selection as an extra baseline at matched $\epsilon$. We expect it to lose substantial accuracy, and if it does not, it is the simpler answer.
Proposed Experiments
We will run on MNIST, CIFAR-10, CIFAR-100, and Purchase100, matching Olive's suite, and add a ViT-Small on CIFAR-100 and a GPT-2-small-scale language model to reach the large-$d$ regime that ERIS motivates at 1.3B parameters. We use 100 clients, label-skewed splits, sparsity from 0.1% to 5%, and total $\epsilon$ from 1 to 8. Metrics are test accuracy, aggregation wall-clock time and enclave memory, label-inference accuracy, and membership-inference AUC from traces.
The first decisive check is whether empirical attack advantage stays under the bound implied by $\rho_{\text{idx}}$, for both the Olive attack and the learned classifier, at page and cache-line granularity. If any attack exceeds the bound, our trace model is wrong, which is precisely SNPeek's warning.
The second is the $d$-sweep. Our aggregation time should stay roughly flat in $d$ while Olive's grows. The crossover point is the paper's practical result.
The third is the budget split. We will plot accuracy against the fraction of $\epsilon$ assigned to the trace, to show whether the side channel can be paid for with a small slice of the budget, say under 20%, without measurable accuracy loss relative to FL-TOP and Olive at the same total.
Where an SEV-SNP machine is available, we will repeat the attack check there. The goal is to see whether ciphertext side channels expose anything beyond the coordinate histogram we account for.
References
- OLIVE: Oblivious and Differentially Private Federated Learning on Trusted Execution Environment
- Olive: Oblivious Federated Learning on Trusted Execution Environment Against the Risk of Sparsification
- SNPeek: Side-Channel Analysis for Privacy Applications on Confidential VMs
- Doquet: Differentially Oblivious Range and Join Queries with Private Data Structures
- Constrained differentially private federated learning for low-bandwidth devices
- EdgeDetect: Importance-Aware Gradient Compression with Homomorphic Aggregation for Federated Intrusion Detection
- Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning
- ERIS: Enhancing Privacy and Communication Efficiency in Serverless Federated Learning