When the Language Model Undoes the Key: Private Split Inference for Vision-Language Models

A vision-language model (VLM) is a large language model that has learned to read images. It works because a small "projector" network turns the output of an image encoder into vectors that the language model treats like extra words. This design makes a privacy scheme look easy. A phone runs the image encoder and a key-scrambled projector, the cloud runs the large language model, and the raw photo never leaves the device. Image cryptography supplies cheap, well-studied ways to scramble such signals: keyed shuffling of blocks, keyed sign flips, and format-preserving Feistel ciphers. Each of these tolerates downstream neural processing on single-task vision models.

We argue this obvious scheme is mostly cosmetic, and that the reason is specific to VLMs. The cloud holds a frozen language model that must understand the scrambled tokens. Whatever it understands, it will soon represent in its own key-independent internal format. It will also describe that content to anyone who asks it to. The proposal has two parts. First, we measure at which layer of the language model keyed visual tokens become key-invariant, and whether an image-reconstruction attacker trained with no knowledge of the key succeeds from that layer on. Second, we test a different defense: a small on-device compressor that reads the user's question and sends only the visual tokens that question needs. The goal of this second defense is to limit what the cloud can learn about the image beyond the answer. It does not hide the answer itself.

Core Claim

In split VLM inference the privacy that matters is set by what the transmitted tokens contain, not by how they are encoded. A secret key applied before a frozen language model protects only the input vectors, and only against an attacker who works at that point in the network. Two stronger attacks are always available to the cloud. One is to read hidden states a few layers into the model. The other is to prompt the model to describe the image. We expect both to recover content regardless of the key.

What does reduce disclosure is a question-conditioned bottleneck on the device. It should beat a question-blind bottleneck with the same token budget on "off-query leakage," meaning how much the cloud can recover about image content the user did not ask about. It should do so at matched answer accuracy.

Image-specific cryptography plays two roles here. It is the set of keyed transforms under test. It also supplies the attack framing: known-plaintext attacks, and attacks that do or do not transfer across keys. The VLM is the application setting. The projector is where the defense lives.

Where This Sits

The work closest to "encrypted images, but machines can still read them" stays unimodal. Human-Imperceptible, Machine-Recognizable Images shuffles image patches with a key and removes position embeddings so a ViT still classifies well, at 79.7% versus 81.8% ImageNet top-1. Detection drops further under its mixing scheme, from 36.1 to 32.9 AP on COCO. EViT encrypts JPEG coefficient codes so that bit-length features stay key-invariant, which lets a cloud retrieve images without the key. Both motivate our question, since neither feeds a language model.

EViT also exposes the core tension. Features that are invariant to the key are exactly the features an eavesdropper can invert without the key. The block-wise encryption studies (ConvMixer transferability, transferability between encrypted models) supply our keyed transforms (SHF block shuffling, NP sign flipping, FFX Feistel encryption). They also supply a key result: attacks transfer at 100% under the same shuffle key and at 25.84% under a different key. That result suggests a key can block transfer when the attacker lives at the model input. Our hypothesis is that inside a VLM the attacker need not live there.

On the VLM side, Deep Pre-Alignment reports that shallow projectors leave visual tokens unaligned, so early language-model layers spend their depth converting them into the model's own format. This is the mechanism we expect to erase the key. DPA also supplies our on-device compressor: the language blocks of a small model (Qwen3-0.6B) placed before the projector. ABC injects text instructions into visual feature extraction; we reuse this idea to make compression question-dependent.

MiniGPT-v2 merges four adjacent visual tokens into one, a 75% reduction. We use this as the question-blind compression baseline. Scalable VLM training via data curation motivates training only the projector while both backbones stay frozen, which is our training regime.

Two papers constrain the design. FIA-Flow reconstructs images from intermediate split-network features, reaching 71.3% classification accuracy on the reconstructions from ResNet-50 features with as few as 128 pairs. It bypasses noise defenses such as Noise+NoPeek and DISCO. We therefore treat additive noise as a baseline, not as a candidate, and use FIA-Flow as our main attacker. PSIC is the nearest crossing of cryptography and VLMs. It blinds contrastive VLMs so they misread images, with 80.8% and 72.3% retrieval attack success. Our goal is the opposite: an authorized generative VLM that answers correctly while seeing as little as possible.

Research Program

The system has three parts. On the device, a ViT encoder feeds a small compressor, either DPA-style Qwen3-0.6B blocks or a resampler, that outputs $m$ tokens. A projector then maps those tokens into the input space of a frozen cloud language model. We use a Qwen3 model at 4B scale, with 32B if budget allows, and Vicuna-7B through a LLaVA-style codebase such as llava-phi as a second backbone.

We train three families of variants.

The keyed variants apply an image-cipher transform $T_k$ with secret key $k$ to the token grid. The transforms are a token permutation (the RS/SHF analog), channel sign flips (the NP analog), and a Feistel-style keyed mixing of channel halves (the FFX analog). The projector is then trained per key so that the frozen language model still answers well.

The question-conditioned variants feed the user's question into the compressor. They are trained with the objective

$$\min_{\theta}\; \mathbb{E}\big[\,\mathcal{L}_{\text{task}}(\mathrm{LLM}(P_\theta(x,q),\,q),\,a)\,\big] \;-\; \lambda\, \mathbb{E}\big[\,\mathcal{L}_{\text{rec}}(A_\phi(P_\theta(x,q)),\,x)\,\big],$$

where $P_\theta$ is the on-device compressor-plus-projector producing $m$ tokens from image $x$ and question $q$, $a$ is the reference answer, $A_\phi$ is a reconstruction adversary trained in alternation, and $\lambda \ge 0$ trades answer quality for privacy. At $\lambda=0$ the only privacy pressure is the token budget $m$ combined with question conditioning.

The analysis has three parts. The first is a layerwise key-invariance study. For each layer $\ell$ of the frozen language model, we compare hidden states produced under key $k$ with those produced under an unkeyed public pipeline. We measure representational similarity between the two. We also measure whether an FIA-Flow inverter trained on unkeyed layer-$\ell$ states transfers to keyed ones.

The second is an attacker suite graded by what the cloud knows. It includes a key-agnostic input-token inverter, the hidden-state inverter above, and FIA-Flow with $n$ known image–token pairs under the victim's key, for $n$ from 0 to a few thousand. It also includes an "oracle" attack, in which the cloud prompts its own language model with the received tokens to describe everything visible.

The third is an off-query leakage probe built on COCO. The question concerns one object, and the attacker tries to recover the presence, attributes, or identity-relevant details of other objects. For identity we use CelebA attributes.

Answer quality is measured on OKVQA, VSR, RefCOCO and MMMU. VSR and RefCOCO also test whether token permutation breaks spatial reasoning, as the COCO drop reported by Hao et al. would predict. Reconstruction is scored with FIA-Flow's metrics: classification accuracy of reconstructed ImageNet images, and LVLM-C and LVLM-PL semantic consistency. The oracle attack is scored against Flickr8k reference captions.

Baselines

The "do nothing" reference is the plaintext split pipeline: a standard ViT, an MLP projector, and the frozen language model, with all visual tokens sent to the cloud. It fixes the utility ceiling and the maximum leakage. For utility on a comparable generalist system, the file reports MiniGPT-v2 at 56.9% on OKVQA (MiniGPT-v2) and a 1.9-point gain for DPA-Qwen3-4B over a LLaVA-style baseline averaged over eight benchmarks (Deep Pre-Alignment).

For the attacker, the strongest reported number is FIA-Flow's 71.3% reconstruction classification accuracy from ResNet-50 features on ImageNet (FIA-Flow). We found no reported result on Lacuna for inverting VLM projector outputs or language-model hidden states, so establishing that number on the plaintext pipeline is itself a deliverable.

The naive privacy baselines are client-side downsampling or blurring, and Gaussian noise on transmitted tokens. They represent the defense family that FIA-Flow already bypasses.

The strongest competitor on our specific question is question-blind compression at the same token budget $m$. We run it three ways: MiniGPT-v2's four-token merging, a learned resampler trained with the identical objective at $\lambda=0$ but without the question, and training-free token compression via VisionTrim. This is the ablation that isolates the proposal's key ingredient.

The keyed image-cipher variants (SHF, NP and FFX analogs) form the second competitor family. They test whether cryptographic keys add anything beyond the bottleneck.

All methods share the same frozen language model, the same instruction data, the same trainable-parameter budget on the device, the same number of optimizer steps, and the same hyperparameter search budget. Attackers get identical data and compute against every defense.

Beating the baselines has a precise meaning. At answer accuracy within one point of the question-blind resampler at equal $m$, the question-conditioned compressor must lower off-query leakage and FIA-Flow reconstruction accuracy by a margin exceeding seed variance. It must also do so across at least two token budgets.

Why not just…?

Why not just decrypt inside a trusted server or secure enclave? If the user trusts the hardware and the operator, that is the correct answer and this work adds nothing. The file's cloud-privacy literature (EViT) targets an untrusted operator. That is our threat model, and we state it explicitly rather than claim to dominate trusted execution.

Why not just run the model under homomorphic encryption? Encrypted arithmetic protects everything, including the answer. However, Hao et al. note that it is too expensive for deep learning at scale, and a multi-billion-parameter autoregressive model is far beyond that regime. We do not test this alternative. We treat it as the confidentiality gold standard that our approach, which only limits disclosure, does not reach.

Why not just scramble the visual tokens with a key, as image ciphers do? This is the obvious idea, and our central experiment exists to test it. The cross-key transfer drop reported by the encrypted-model study is real evidence that keys can block input-level attacks. However, DPA's finding that early layers perform format alignment predicts that the frozen language model will map keyed tokens back into a shared internal format. The oracle attack needs no inverter at all. If the layerwise study shows keyed hidden states remaining non-transferable deep into the model, and the oracle attack recovers less off-query content under keys, this alternative wins. We will then report it as the method.

Why not just send fewer visual tokens? This is the most serious threat, because a small $m$ already discards information. It may explain most of any privacy gain. The matched-budget comparison against question-blind compression settles it. If question conditioning gives no leakage reduction at equal accuracy, the proposal reduces to "compress hard." We would report that as the finding.

Why not just add noise or differential privacy? FIA-Flow bypasses Noise+NoPeek and DISCO. Hao et al. note that differential privacy substantially degrades model performance. We include noise only as a reference curve.

Decisive Experiments

Three results decide the proposal.

The first is the layerwise key-invariance curve. For each keyed transform, we plot inverter transfer and hidden-state similarity against language-model depth. If an unkeyed inverter reaches near-plaintext reconstruction accuracy from layer-$\ell$ states for small $\ell$, image-style keys are cosmetic in VLMs. That is a clean negative result the field does not yet have. Our secondary check sweeps the known-plaintext pair budget $n$ and asks how few pairs FIA-Flow needs to break each key. The file's figure of 128 pairs for plain features sets the reference.

The second is the matched-budget comparison of off-query leakage between question-conditioned and question-blind compression on COCO and CelebA probes, holding OKVQA, VSR and MMMU accuracy fixed.

The third is the oracle attack on the best configuration, reported alongside plaintext. It states honestly how much a curious cloud still learns simply by asking its own model. That number marks the practical ceiling of any split-inference privacy scheme built on a frozen language model.