Borrowed Blind Spots: Keyed Null-Space Obfuscation for Split Vision Transformers Without Server Gradients

Split inference runs the first few layers of a network on a user's device and sends the intermediate activations to a server, which runs the rest. With vision transformers (image models that cut a picture into patches and process them as a sequence of tokens), this is risky. Unlike convolutional networks, vision transformers do not shrink the image as it moves through the layers, so the tokens sent to the server keep a great deal of pixel detail. DRAG shows that a pretrained diffusion model can rebuild recognizable private images from vision transformer tokens as deep as layer 12, and that standard fixes such as channel pruning and token shuffling do not stop it. For language models, OSNIP found a way out. High-dimensional representations contain directions the downstream model ignores, a "semantic null space," so a client can push its embeddings far away geometrically without changing the model's output. OSNIP reports 100.13% utility preservation and a 0.00% nearest-neighbor attack success rate on Qwen3-32B. The catch is that training OSNIP's client-side encryptor needs gradients from the server's model. A privacy-conscious client facing a proprietary server will not have those gradients.

This proposal asks whether a client can borrow the blind spots of a public model instead. The idea comes from work on adversarial transferability, meaning perturbations computed on one model that also fool another. Fine-tuned models inherit the geometry of their public pretrained parents (Two Sides of the Same Coin), and published compressed models give attackers a surrogate that transfers to the server-side parent (Publishing Efficient On-device Models Increases Adversarial Vulnerability). Attackers have treated this shared geometry as a weakness to exploit. We propose that it can also protect users. If the directions a fine-tuned server tail ignores are largely the same as those its public parent ignores, a client can train a keyed obfuscator against the public parent alone, and the server's clean accuracy will survive while reconstruction fails.

Core Claim

Consider the common deployment where the first $k$ layers (the "head") come from a public pretrained vision transformer and the server fine-tunes the remaining layers (the "tail") privately. We claim three things. First, the near-null directions of the server's private tail overlap strongly with those of the public tail it was fine-tuned from. Second, an obfuscator trained only through the public tail therefore keeps the private server's clean accuracy close to undefended levels. Third, training transfers better when the obfuscator is kept from fitting quirks of the surrogate's deep attention layers. This mirrors how Adaptive Token Tuning improves attack transfer by truncating deep-layer attention gradients.

The prediction can be checked directly. Let $J_s$ and $J_t$ be the Jacobians of the surrogate and target tail logits with respect to the layer-$k$ tokens. Let $N_s$ and $N_t$ be the spans of their bottom singular directions. The overlap is

$$\rho = \frac{1}{r}\left\| P_{N_s} P_{N_t} \right\|_F^2,$$

where $P$ denotes orthogonal projection onto each span and $r$ is the span dimension. $\rho$ equals 1 when the null spaces coincide and is near 0 when they are unrelated. We predict that $\rho$ will predict how much clean accuracy the obfuscator keeps on each target.

The novelty is not null-space obfuscation itself, which is OSNIP's contribution. It lies in removing OSNIP's need for server gradients by treating defense transfer as a transferability problem. It also lies in testing this on vision transformers, where OSNIP explicitly leaves extension open.

Setting and Threat Model

The client holds the public head and a public copy of the tail. The server holds a private tail, fine-tuned from a public checkpoint the client may or may not know. The server is honest-but-curious: it computes the requested prediction but also tries to reconstruct the image or infer private attributes.

Following Kerckhoffs's principle, the attacker knows the obfuscator's architecture and training procedure but not the per-session key. It may therefore train a reconstructor on obfuscated tokens generated with keys it samples itself. This adaptive attacker is what defeated token shuffling in DRAG, where a small network learned to undo the permutation. Any defense that only works against a non-adaptive attacker counts as a failure here.

Research Program

We would build a small client-side MLP obfuscator. It takes layer-$k$ tokens and a random key and outputs tokens of the same shape. Its training objective follows OSNIP's three parts, applied to vision tokens:

  • a utility term, the KL divergence between the surrogate tail's predictions on clean versus obfuscated tokens;
  • a privacy term, a hinge loss that pushes obfuscated tokens toward near-zero cosine similarity with the clean ones;
  • a diversity term, which makes outputs depend on the key so that no single inverse map works.

The new ingredient is transfer-aware training of the utility term. When the utility gradient passes back through the surrogate tail, we apply Adaptive Token Tuning's hybrid truncation, zeroing attention-path gradients in deep layers and rescaling them in shallow ones. We also randomly drop patches in the style of its Self-Paced Patch Out. The intent is for the obfuscator to rely on directions shared across fine-tunes rather than on idiosyncrasies of the surrogate. A second variant replaces the single surrogate with a small ensemble of public checkpoints at the same width, such as differently pretrained ViT-B/16 models. This tests whether averaging over parents works better than gradient smoothing.

The analysis has two parts. We compute $\rho$ between the public tail and each private target and correlate it with the clean-accuracy gap. We also map how far fine-tuning can drift before transfer breaks, by varying fine-tuning length, learning rate, and how many tail layers are unfrozen.

The scope is deliberately narrow. Cross-architecture transfer, for example to a convolutional server, is out of scope, because the tail must accept the head's token format. The question is transfer across private tails that share a head, which is the realistic deployment.

Baselines

The naive reference is undefended split inference. It sets the clean accuracy to match and the reconstruction quality an attacker reaches with no defense. Next to it is Gaussian noise added to the transmitted tokens at matched clean accuracy, which stands in for differential privacy.

The strongest existing competitor run under the same constraints is adversarial representation learning against a surrogate inverter, of the NoPeek type. Here the obfuscator is trained to fool a reconstructor the client trains through the same public surrogate. It is the natural reviewer alternative, and DRAG reports that NoPeek-style defenses fall to diffusion attacks. It is run on the same data and surrogate, with the same obfuscator capacity and the same tuning budget, measured in GPU-hours and hyperparameter trials.

The oracle upper bound is OSNIP-style training with white-box access to the private target tail. It shows how much of the oracle's privacy-utility frontier the gradient-free method recovers. The closest published numbers come from other settings and cannot be compared directly:

  • OSNIP reports 100.13% utility preservation and 0.00% nearest-neighbor attack success rate on Qwen3-32B, a language model.
  • ADP reports about 4% utility loss and multi-scale structural similarity (MS-SSIM) of reconstructions dropping from 0.80 to 0.18 on CelebA, using convolutional backbones.
  • DP-CutMixSL reports 73.77% CIFAR-10 accuracy with a trusted mixer.

We will rerun ADP's autoencoder on the vision transformer split for a direct comparison. We found no reported result on Lacuna for any defense against DRAG on vision transformer tokens, nor for DRAG's own MS-SSIM or LPIPS values, so the undefended DRAG numbers will be produced here.

Two ablations isolate the key ingredients. The first removes transfer-aware gradient truncation, which isolates the borrowed Adaptive Token Tuning mechanism. The second is an untrained, keyed random perturbation projected onto the surrogate Jacobian's estimated null space. It tests whether learning adds anything beyond linear algebra.

"Beating" a baseline has a specific meaning. Against the adversarial-inverter baseline, at matched clean accuracy on the private target, the method must produce lower adaptive-DRAG MS-SSIM and higher LPIPS (a learned perceptual distance), with non-overlapping seed intervals. Against noise, the method must keep at least the clean accuracy that noise keeps while reaching attack scores noise cannot reach at any level. The method must also stay within ADP's roughly 4% utility loss on the private target.

Why not just…?

Why not just add noise to the tokens? The evidence against this is strong. Seeing the Forest through the Trees finds that differential-privacy noise large enough to stop transformer leakage drives task accuracy to near zero. DRAG finds that noise on vision transformer activations still yields recognizable reconstructions. We still run noise at matched accuracy, so the gap is measured in our setting rather than assumed.

Why not just train the obfuscator against a learned inverter on the surrogate? This is the real threat to the proposal. If it matches our method under adaptive diffusion attack, the null-space framing adds nothing, and we will report that. Our reason to expect a gap is that this baseline overfits to one inverter, in the same way attacks overfit to one surrogate classifier. DRAG shows a stronger diffusion prior defeats such defenses. The head-to-head comparison at equal budget, scored by adaptive DRAG rather than the training inverter, settles the question.

Why not just ask the server for gradients and use OSNIP directly? When the server cooperates, it should, and we include that as the oracle. The proposal targets the case where the server is the adversary or will not expose its weights. That is the limitation OSNIP itself names. Its value is measured by how close it gets to the oracle.

Why not just use fixed position embeddings or shuffle tokens? APRIL shows that fixed position embeddings close one gradient channel, but DRAG reconstructs from activations without that channel, and it undoes shuffling with a learned reordering network. We include shuffling as a sanity baseline under the adaptive attacker.

Proposed Experiments

The main grid uses ViT-B/16 with splits at layers 4, 8, and 12 on ImageNet-1K. The public surrogate is the pretrained checkpoint. Private targets are tails fine-tuned on CIFAR-10, ImageNet subsets, and CelebA attributes, at increasing drift from the parent.

Privacy is measured three ways:

  • reconstruction by DRAG with Stable Diffusion guidance, run both non-adaptively and adaptively with attacker-sampled keys, scored by MS-SSIM and LPIPS on ImageNet-1K, MSCOCO, and FFHQ;
  • attribute inference on CelebA, following ADP's protocol of predicting a secondary attribute when the task is a different one;
  • nearest-neighbor matching of obfuscated tokens to clean tokens.

Utility is the private target's clean accuracy relative to its undefended accuracy. Three seeds per configuration.

Three checks are decisive:

  1. Adaptive attack. If adaptive DRAG with attacker-sampled keys brings reconstruction quality back near undefended levels, the keyed obfuscation fails and the claim is refuted.
  2. Overlap analysis. If $\rho$ does not predict the clean-accuracy gap across targets, the "shared blind spots" explanation is wrong, even if some configurations happen to work.
  3. Truncation ablation. If removing transfer-aware truncation does not reduce accuracy on the private targets while leaving it unchanged on the surrogate, then the transferability mechanism contributes nothing. In that case the result reduces to OSNIP ported to vision.