Hide, Don't Scrub: A Small Encrypted Channel for Split Inference

When a phone or camera is too weak to run a full neural network, it can run the first few layers itself and send the intermediate numbers to a cloud server, which finishes the job. This is called split inference. The catch is that those intermediate numbers can often be turned back into the original photo. Current defenses respond by training the device-side layers to throw away whatever an attacker could use to rebuild the image. Throwing information away costs accuracy, and these defenses are usually tested only against the attacker they were trained against. We propose a different route. The device sends two things. The first is a large plaintext feature that has been trained to be hard to reconstruct from. The second is a tiny dense vector, encrypted with homomorphic encryption (a kind of encryption that lets the server compute on data it cannot read), which carries the fine detail the task still needs. The server combines the two inside a shallow polynomial head that never has to decrypt anything. Because the encrypted channel is small and the encrypted computation is shallow, the whole exchange takes one round trip and needs no bootstrapping, the costly ciphertext-refresh step of homomorphic encryption.

The bet is that "hide" beats "scrub." Information the task needs but an attacker could exploit does not have to be destroyed if it can be moved into a channel whose secrecy rests on cryptography rather than on the attacker being weak. If the bet holds, an attacker can rebuild the input only as well as the plaintext channel allows. We then test that plaintext channel against attackers who know the defense and adapt to it, including data-free layer-by-layer inverters, and we measure whether the accuracy that scrubbing defenses give up comes back.

Core Claim

At a fixed privacy level, measured against attackers who adapt to the defense and were never seen during training, a split model with a learned plaintext/encrypted division of labor reaches higher task accuracy than the best plaintext obfuscation defenses. It does this at a client cost close to plaintext split inference: one encryption of one small vector, one decryption of the logits, one network round. A server-side homomorphic tail would cost far more. The new ingredient is the routing. The device-side network is trained so that whatever it leaves in plaintext resists reconstruction, while the encrypted vector is allowed to carry anything at all.

Positioning Against Prior Work

Several papers motivate the problem by showing that plaintext split features leak. Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks introduces DCI, which inverts the network block by block using synthetic data matched to the network's own batch-normalization statistics. It recovers recognizable ImageNet images from features up to 42 convolutional layers deep, so cutting the network later is not a defense on its own. Variational Model Inversion Attacks shows that a StyleGAN prior paired with normalizing flows gives far stronger inversions than earlier generative attacks (55% vs. 7% attack accuracy on CelebA). We repurpose it as a feature-space attacker, which that paper did not evaluate. Optimizing Adaptive Attacks against Content Watermarks for Language Models and the critique of mask-style defenses such as DeepCloak supply the methodological lesson: a defense must be judged against an adversary who optimizes against it.

Four plaintext defenses constrain what we must beat. PATROL prunes the device model so deeper layers fit on the device, adds Lipschitz regularization, and trains against a surrogate inverter. ResSFL pre-trains a reconstruction-resistant extractor with a narrow bottleneck layer and transfers it to clients. InfoDecom removes low-frequency image content, applies an information bottleneck, and adds calibrated noise. Leveraging Sparsity for Privacy in Collaborative Inference uses a device-side sparse autoencoder with noise on both which coordinates are active and their values. All four protect by removing or perturbing information in a plaintext feature, and none is evaluated against DCI-style or architecture-shifted adaptive inverters. We reuse the sparse autoencoder from the last of these as our plaintext channel. We differ by adding a cryptographically protected side channel, so that removal pressure no longer has to be traded directly against accuracy.

Three cryptographic lines supply the mechanism and its limits. Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption (SAFHIRE) avoids bootstrapping by having the client decrypt, apply each nonlinearity, and re-encrypt at every layer. That requires many network rounds and an always-online client. Its shuffling protects the server's weights, not the client's input. We keep the "no bootstrapping" goal but get it from shallowness instead of interaction. Activation-Free Backbones for Image Recognition (PolyNeXt) shows that multiplication-based blocks of the form $W_o((W_a x)\ast(W_b x))$ reach 83.9% ImageNet top-1 without pointwise activations, which makes them natural for encrypted evaluation. We use a one-block version as the encrypted head. That paper never deployed under encryption and never studied invertibility. Evaluating Selective Encryption Against Gradient Inversion Attacks shows that encrypting 10–40% of the largest gradient entries stops fixed gradient-inversion attacks. It does so for federated gradients, not forward features, and with a heuristic selection rule rather than a learned one. That heuristic becomes our key ablation. Salted Inference hides the server's output with a secret label permutation but leaves inputs invertible. Our encrypted logits give a stronger form of output hiding as a side effect, though only partially, as discussed below.

Research Program

The device runs the first $L$ layers $f_\theta$ of a backbone. It then produces two outputs from the shared feature. The first is a sparse plaintext code $p = S(f_\theta(x))$ from the sparse autoencoder, with its index and value noise kept as published. The second is a small dense vector $h = R(f_\theta(x)) \in \mathbb{R}^{d_e}$, with $d_e$ between 16 and 512. The vector $h$ is packed into a single CKKS ciphertext. CKKS is an approximate-arithmetic encryption scheme built on ring learning-with-errors, and one ciphertext holds thousands of slots, so the client's encryption cost does not grow with $d_e$ in this range. Because $h$ is dense and fixed-length, its shape reveals nothing.

The server runs the remaining plaintext backbone on $p$ to obtain a feature $u$. It then evaluates one polynomial block on the mixed input:

$$z = W_o\big((A u + B\,\mathrm{Enc}(h)) \ast (C u + D\,\mathrm{Enc}(h))\big),$$

where $A, B, C, D, W_o$ are server weights and $\ast$ is elementwise product. The output $z$ is the vector of class logits, still encrypted. The block needs plaintext-by-ciphertext multiplications, one ciphertext-by-ciphertext multiplication, and a rotation-based matrix product, so a multiplicative depth of about three suffices and no bootstrapping is required. The encrypted logits return to the client, which decrypts them locally.

Training uses a defense-aware min-max objective. The adversary term applies only to the plaintext channel:

$$\min_{\theta,S,R,\text{server}} \; \mathcal{L}_{\text{task}}(z, y) \;-\; \lambda \,\min_{\phi \in \Phi_t} \mathcal{L}_{\text{rec}}\big(G_\phi(p), x\big) \;+\; \mu\, d_e .$$

Here $\mathcal{L}_{\text{rec}}$ is a reconstruction loss (MSE plus LPIPS), and $G_\phi$ is an inverter drawn from a family $\Phi_t$ that rotates during training among a decoder network, a DCI-style block-wise inverter, and a generative-prior inverter. Rotating the family is meant to avoid overfitting to a single surrogate attacker, which is the failure mode attributed to PATROL. The coefficient $\lambda$ weights privacy pressure on the plaintext channel, and the term $\mu\, d_e$ prices the encrypted bandwidth. The training is made differentiable by simulating the encrypted head in plaintext, with CKKS-like approximation noise injected.

The security argument splits cleanly in two. Under the standard chosen-plaintext security of CKKS, the server learns nothing from $\mathrm{Enc}(h)$, and the client never returns decrypted values to it. The attacker's reconstruction is therefore bounded by what can be extracted from $p$ alone, together with full knowledge of the architecture and the device weights. Everything the cryptography does not cover becomes an empirical question about $p$, and that is where we focus the adaptive evaluation.

We will not claim output privacy. The server sees $u$ and can train its own classifier on it. We will measure how much class information $u$ retains and report it plainly, rather than asserting that the encrypted logits hide the label.

Baselines

The naive reference point is undefended split inference with the cut at the same layer $L$. It anchors the reconstruction numbers and shows how much any defense buys.

The strongest plaintext competitor is the sparse-autoencoder defense of Leveraging Sparsity for Privacy in Collaborative Inference. It already has a tiny payload and the best reported privacy-utility point in the file: 89.50% accuracy at reconstruction MSE 0.0231 on FaceScrub, with payloads under 1% of the original size. It is also literally our plaintext channel, so the comparison isolates the encrypted side channel. InfoDecom (CIFAR-10: accuracy 0.7329 at reconstruction MSE 0.0843), ResSFL (CIFAR-100: 67.5% accuracy, reconstruction MSE 0.050 vs. 0.005 undefended), and PATROL (VeRi: PSNR down 21.5% and SSIM down 20.1% at a 12.7% accuracy cost) are rerun as additional plaintext baselines.

The reported numbers come from each paper's own attacker and may use different normalization. We therefore re-run every baseline with its released recipe and our shared attacker suite. All methods get the same backbone, the same cut layer, and the same number of tuning trials over their privacy coefficients. We then compare privacy-utility frontiers rather than single points. We have no reported result found on Lacuna for any of these defenses under DCI or under generative-prior feature inversion; producing those numbers is itself a contribution.

The strongest cryptographic competitor is SAFHIRE, which reports a 13.65 s CIFAR-10 ResNet-20 inference on an A100 at 1.25 MB/s and accuracy within 2.1% of ORION. We run it on the same network and simulated link, counting client CPU time, energy, rounds, and bytes. A second cryptographic reference encrypts the entire device output and evaluates the whole server tail under leveled encryption with polynomial activations, using a PolyNeXt-style tail and bootstrapping only where depth forces it. This is the "encrypt everything" ceiling.

Two ablations isolate the key ingredient. Setting $d_e = 0$ collapses our method to the plaintext sparse defense trained with our rotating attacker family, which separates the benefit of the attacker rotation from the benefit of encryption. Replacing the learned encoder $R$ with a fixed rule encrypts the largest-magnitude sparse coordinates, the "Grad" rule from Evaluating Selective Encryption Against Gradient Inversion Attacks, at the same ciphertext budget. That tests whether learning the routing matters.

Beating the strongest baseline means two things. First, at equal or better attack-side metrics under the held-out adaptive suite (reconstruction SSIM and LPIPS, and the identification accuracy of reconstructed faces), we exceed the sparse defense's accuracy by a margin larger than seed variance on FaceScrub and CIFAR-10. Second, client latency and energy stay within a small constant factor of plaintext split inference, rather than the many rounds SAFHIRE needs.

Why not just…?

Why not just encrypt the whole feature and run the rest of the network homomorphically? This gives cryptographic input privacy with no plaintext leakage at all. Its cost lives on the server: the deep tail must use polynomial activations and, beyond modest depth, bootstrapping. Circuit Bootstrapping: Faster and Smaller identifies bootstrapping as the central cost, and A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification reports tens of gigabytes of traffic for a single convolutional-network inference. This is a real threat for small tails and shallow models. Our crossover experiment varies tail depth and reports where the hybrid stops being cheaper. If the crossover sits at depths smaller than the ones real devices need, we will say so.

Why not just train a plaintext defense harder, with stronger attacker-aware training? If the plaintext defenses, once trained against a rotating attacker family, already hold up against held-out attackers at near-undefended accuracy, the encrypted channel is unnecessary. The $d_e = 0$ ablation, given the same training budget and the same attacker rotation, settles this directly. DCI's depth results give a prior that plaintext features keep enough structure to be inverted, but we treat this as the main open risk rather than a settled point.

Why not just use SAFHIRE's interactive protocol, which keeps exact ReLUs? SAFHIRE needs one client decrypt-and-re-encrypt per nonlinear layer, and its shuffling protects server weights, not client inputs. On a battery-powered device with an unreliable link, the round count is the cost. We measure rounds, client energy, and failure under packet loss for both systems on CIFAR-10 ResNet-20.

Why not just encrypt the biggest coordinates, as selective gradient encryption does? That rule was validated only against fixed attacks. An adaptive inverter can treat the encrypted coordinates as missing and train on the rest. The magnitude-routing ablation, attacked by inverters trained on the visible coordinates only, tells us whether the learned encoder is doing real work.

Why not just run everything on the device? Deep Private-Feature Extraction measured about 120% extra latency on a Nexus 6P from a few more layers, and providers will not ship proprietary server weights to clients. We report fully local latency as a reference, but it is outside the regime this work targets.

Proposed Experiments

We evaluate on CIFAR-10 and CIFAR-100 with ResNet-18 and VGG-11, on FaceScrub as the face-privacy stress test, and on VeRi to match PATROL's surveillance setting. Cut points are chosen to match the published baselines. Client timing and energy are measured on a Raspberry Pi 4-class board and an Android phone running a CKKS library. Server timing is measured on a GPU. Links are simulated at 1.25 MB/s to match SAFHIRE.

The attacker suite is the core of the evaluation. It includes a decoder trained on public auxiliary data; DCI adapted to our plaintext channel and given the device weights; a VMI-style StyleGAN-plus-flow inverter operating on $p$; and, for the final evaluation, attacker architectures and losses never used in training, each fine-tuned against the deployed defense. We report MSE, PSNR, SSIM, LPIPS, and face-identification accuracy of reconstructions. We also report task accuracy, a probe for class information in $u$, client milliseconds and joules, bytes, rounds, and server seconds.

Three checks are decisive. The first is the held-out-attacker frontier: our curve must lie above the sparse-defense curve and the $d_e = 0$ curve. A drop in privacy from training-time to held-out attackers that is as large for our method as for the baselines would mean the gain comes from overfitting to a surrogate, not from hiding information. The second is the $d_e$ sweep: accuracy should rise and saturate while held-out reconstruction quality stays flat, and if reconstruction worsens as $d_e$ grows, the routing is leaking. The third is the depth crossover against the fully encrypted tail, which tells practitioners exactly when the small encrypted channel is the right tool and when they should simply encrypt everything.