Search engines, retrieval-augmented chatbots and on-device assistants turn text into embeddings, fixed-length vectors that capture meaning, and then send those vectors to servers or store them in vector databases. Engineers have long assumed a vector cannot be turned back into its sentence. Recent attacks show the assumption is wrong. A decoder trained to "translate" vectors back into text recovers 92% of 32-token inputs exactly (Text Embeddings Reveal (Almost) As Much As Text). A decoder built on a stolen stand-in encoder recovers patient sex and age from clinical-note embeddings at about 99% accuracy (Transferable Embedding Inversion Attack). Current defenses treat the encoder as fixed and add noise or masks to its output afterward. Hardware-aware neural architecture search is an automated way to choose layer depths, widths and attention heads so that a network meets a latency or memory budget on a given device. It already chooses the architecture of these encoders when they run on phones and small CPUs, but it scores candidates only on accuracy and speed.
This proposal asks a question neither field has tested: at fixed retrieval quality and fixed on-device latency, do different encoder architectures leak different amounts of recoverable text? If they do, then the same search that makes an encoder fast on a Raspberry Pi can also make it harder to invert, at no extra runtime cost. That matters because the output-side defenses add client computation, which is exactly what edge deployment cannot afford. The first half of the project is a measurement study designed so that a "no" answer is informative and publishable. The second half builds the search only if the measurement says architecture matters.
Core Claim
We claim that encoder architecture changes the trade-off between retrieval utility and inversion leakage, and that the change is not explained by output dimension alone. Output dimension is the obvious confound. Width and depth, how per-token states are pooled into one vector, the shape of the final projection, and uneven layer widths decide how much word-level detail survives into the vector beyond what retrieval needs. We further claim that the architecture with the best trade-off differs by device, because the devices already favor different shapes. HAT showed that GPUs favor wide, shallow transformers while ARM CPUs favor deep, narrow ones.
The search problem we pose is
$$\max_{\alpha \in \mathcal{A}} U(\alpha) \quad \text{s.t.} \quad T_H(\alpha) \le \tau_H,\;\; L(\alpha) \le \delta .$$
Here $\alpha$ is a sub-network drawn from a search space $\mathcal{A}$ of encoder architectures. $U$ is retrieval utility, measured as NDCG@10 on BEIR (a ranking-quality score for retrieval). $T_H$ is measured latency on device $H$ with budget $\tau_H$. $L$ is inversion leakage under an adaptive attacker with budget $\delta$.
The novelty is not adding a third objective to the search. It is establishing whether $L$ varies across architectures once $U$ and output dimension are held fixed, and finding a cheap, trustworthy estimate of $L$ that the search can call thousands of times.
Relation to Prior Work
Two lines of work motivate the proposal. The attack literature shows that leakage is large and survives simple perturbation. ZSinvert (Universal Zero-shot Embedding Inversion) inverts new encoders without per-encoder training, stays effective under Gaussian noise up to σ = 0.01, and exceeds 80% sensitive-information leakage on Enron email. Multilingual encoders leak across languages: an English-trained inverter reaches 80.02 BLEU on Spanish embeddings (Text Embedding Inversion Security for Multilingual Language Models). The SPARSE analysis (Concept-Aware Privacy Mechanisms for Defending Embedding Inversion Attacks) finds that sensitive concepts concentrate in a few "privacy neurons." If an architecture's width and projection decide where such concentration happens, then architecture is a plausible lever. That connection is the hypothesis we test.
A second group constrains the design. Every defense in the file works after the encoder, on a frozen backbone. This includes SPARSE's learned mask with direction-dependent noise, DP-Forward's noise injected during the forward pass (DP-Forward), null-space obfuscation (OSNIP), and first-dimension masking. None of them profile edge latency. We do not replace them. We ask whether a better backbone makes them unnecessary, or leaves them with less work to do.
The closest neighbor across the two fields is GI-NAS (GI-NAS). It searches decoder architectures to strengthen gradient-inversion attacks in vision federated learning. It searches on the attacker's side, in another modality, with no hardware model. We search the defender's encoder, for text, under measured device budgets. Zero-cost Proxy for Adversarial Robustness Evaluation shows that robustness can be scored at initialization. Robustness to input perturbation is a different property from non-invertibility of outputs, though, so we do not expect its NTK proxy to transfer. We test it as a candidate proxy and expect it to fail.
The method comes from HW-NAS. We use HAT's weight-shared "SuperTransformer," a single large network whose sub-networks share weights, with uneven per-layer widths and a learned latency predictor. We use HW-GPT-Bench's sandwich-rule supernet training and its surrogate models of device cost (HW-GPT-Bench). HW-GPT-Bench also found that embedding dimension dominates model quality, which is why dimension must be controlled explicitly. For the leakage bound we use ConNAS's gradient steering (Constraint-guided Hardware-aware NAS through Gradient Modification). ConNAS enforces non-differentiable limits as hard boundaries by nudging the architecture back into the allowed region, rather than adding a weighted penalty to the loss.
Research Program
Encoder supernet. We build a BERT-style encoder supernet. Depth, per-layer hidden width, number of attention heads, feed-forward size, pooling (mean, first-token, attention-weighted) and output projection dimension are all searchable, and output dimension ranges from 128 to 768. It is trained once with contrastive retrieval loss on MS-Marco using the sandwich rule. We profile latency on a Raspberry Pi-4, an Intel Xeon CPU and a consumer GPU, the same device mix as HAT, and fit a latency predictor per device.
Leakage measurement. For about 150 sampled sub-networks spread across output dimensions, we train a fixed-budget adaptive inverter against each one. The inverter is a Vec2Text-style decoder with iterative correction, given query access to that exact sub-network. We also run ZSinvert and the transferable surrogate attack against each. Leakage is recorded as token F1 and exact match on held-out MS-Marco passages, plus sex and age attribute recovery on MIMIC-III.
Weight sharing may distort leakage, since a sub-network's inherited weights are not the weights it would learn alone. To check this, the 20 most informative sub-networks are retrained standalone and re-attacked.
The decisive analysis. We regress leakage on utility, output dimension and latency. We then measure how much of the leftover variation architecture features explain, using held-out sub-networks. If architecture explains a meaningful share, for example if sub-networks with matched dimension and matched NDCG@10 differ by more than 10 token-F1 points under the adaptive attack, we proceed to search. If not, the contribution becomes a careful negative result: leakage is fixed by dimension and utility, so hardware-aware dimension choice is the only architectural lever. That result would still correct the heterogeneity hypothesis raised in the research file.
Cheap leakage estimate. Training an inverter costs too much to run inside a search loop. We therefore test three candidate stand-ins against the adaptive inverter's leakage, using rank correlation (Kendall's τ) across the measured sub-networks. The first is a linear probe that predicts the bag of input tokens from the embedding. The second is a short inverter trained for about 1% of the full budget. The third is the NTK-based robustness proxy. We would accept a stand-in at τ ≥ 0.8, a slightly lower bar than the τ > 0.90 that HW-GPT-Bench reports for its device-cost surrogates.
Constrained search. An evolutionary search, following HAT, samples sub-networks. Latency is enforced with the device predictor, and ConNAS-style boundary steering handles the leakage limit $\delta$. Searched architectures are retrained standalone and attacked with the full adaptive inverter. That full attack is the only leakage number we report as a result; stand-in scores are never reported as results.
Baselines
The naive reference point is the best-utility sub-network of our own supernet under each device's latency budget, found by ordinary HW-NAS with no leakage term, and attacked with the same adaptive inverter. Everything is measured relative to this point. It answers the question "what does ignoring privacy cost?" on identical data, supernet and search budget.
The strongest existing defense is SPARSE, applied on top of that naive HW-NAS encoder, with its client-side overhead measured on each device. The file reports SPARSE lowering STS12 leakage from 60% to 19% while keeping 65% utility at ε = 10, against about 22% leakage and 60% utility for standard DP. On MIMIC-III it lowers sex-attribute leakage from 88% to 28%. Those figures come from off-the-shelf backbones, so we rerun SPARSE on our encoders, using its published ε sweep and the same attacker.
Beating SPARSE means one of two outcomes. The first is a searched encoder, with no output-side mechanism, that matches SPARSE's leakage at equal NDCG@10 with lower on-device latency. The second is searched encoder plus SPARSE strictly dominating naive encoder plus SPARSE on the utility–leakage curve. DP-Forward is the second defense baseline, tuned over its noise scale. The file reports it reducing inversion success by up to 88 percentage points, but not on our benchmarks.
The ablation that isolates the key ingredient keeps output dimension fixed and varies everything else. If the gains vanish when only dimension is searched, then the architecture claim fails even if the search "wins."
A final baseline is adversarial fine-tuning of the naive encoder against a co-trained inverter, with the same GPU budget as our search. No reported result found on Lacuna for adversarially trained anti-inversion text encoders, so we implement it ourselves and tune it with the same budget.
Why not just…?
Why not just choose a smaller output dimension on an existing efficient model? This is the most serious threat, and the project is built around it. If leakage is determined by dimension and utility, then the search adds nothing. The dimension-only ablation and the matched-dimension regression settle the question directly, and a negative outcome is reported as the finding.
Why not just add noise to an efficient model's embeddings? SPARSE reports that noise applied equally in every direction costs utility: STS12 utility falls to 60%, against 65% for its targeted noise. ZSinvert still inverts at σ = 0.01. Noise also cannot move the frontier that the backbone defines. We include DP-Forward and SPARSE as baselines and test whether a searched backbone plus noise beats a naive backbone plus noise.
Why not just fine-tune the encoder adversarially against an inverter? This is also a real threat. Fine-tuning may capture most of whatever the architecture captures, and adversarial training against one decoder often fails against a stronger adaptive one. We run it with equal compute, attack it with a freshly trained inverter the defender never saw, and also test the combination of search and fine-tuning. If fine-tuning alone matches the combination, we will say so.
Why not rely on heterogeneous layers to break transferable attacks? An uneven architecture may defeat ZSinvert and the surrogate attack only because it looks unfamiliar to them. An attacker with query access simply trains against it. That is why the reported leakage number always comes from the adaptive inverter. We report the gap between transfer-attack leakage and adaptive leakage separately, as a measure of false security.
Decisive Checks
The work is evaluated on MS-Marco v2.1 for reconstruction (token F1 and exact match), on MIMIC-III for attribute leakage, on Enron for sensitive-content leakage, and on BEIR and STS via MTEB for utility. Four outcomes decide the project. First, whether architecture explains leakage variation beyond dimension and utility under adaptive attack. Second, whether a cheap stand-in reaches τ ≥ 0.8 with the adaptive inverter. Third, whether the searched Pareto fronts differ across Pi-4, Xeon and GPU. Fourth, whether searched encoders beat naive encoder plus SPARSE in latency at matched leakage and utility.
For scale, an undefended GTR encoder yields 31.81 token F1 under ZSinvert, rising to 54.39 with its correction model. The file reports this for GTR only, so it serves as context rather than a target for our encoders. If the first check fails, the rest of the program stops, and the negative result is the paper.
References
- Text Embeddings Reveal (Almost) As Much As Text
- Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
- Universal Zero-shot Embedding Inversion
- Text Embedding Inversion Security for Multilingual Language Models
- Concept-Aware Privacy Mechanisms for Defending Embedding Inversion Attacks
- DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
- OSNIP: Balancing the Privacy-Utility-Efficiency Trilemma in LLM Inference via Obfuscated Semantic Null Space
- GI-NAS: Boosting Gradient Inversion Attacks Through Adaptive Neural Architecture Search
- Zero-cost Proxy for Adversarial Robustness Evaluation
- HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
- Constraint-guided Hardware-aware NAS through Gradient Modification
- MTEB: Massive Text Embedding Benchmark