Benign to Train, Trapped to Invert: Landscape Engineering for Private Low-Rank Fine-Tuning

In federated learning, clients never send raw data to the server; they send gradients. But a gradient is a deterministic function of the data that produced it, and an eavesdropper can run an attack that searches over candidate inputs until the gradients they induce match the observed ones — often recovering training images or text with high fidelity. A popular way to cut communication and computation is LoRA-style fine-tuning, where the model's weights stay frozen and only a small update, written as the product of two thin matrices, is trained and shared. This creates two optimization problems sitting side by side: the defender's problem, minimizing training loss over the low-rank factors, and the attacker's problem, minimizing the mismatch between observed and synthetic gradients over candidate data. We know a great deal about the geometry of the first problem — under reasonable conditions every local minimum of the factorized training loss is actually a global minimum — and almost nothing about the geometry of the second.

This proposal characterizes the attacker's optimization landscape when updates are low-rank factorizations, and then exploits the asymmetry between the two landscapes as a defense. The idea in two sentences: choose the factorization's rank and structure so that the defender's training landscape stays benign (every local minimum global, every other stationary point a saddle that noisy gradient descent escapes), while the attacker's inversion landscape becomes hard — its set of gradient-matching solutions swells into a high-dimensional manifold of wrong answers, and its optimization runs stall in spurious local minima that absorb most random restarts. We then measure where this "landscape defense" sits on the privacy-utility frontier, using real reconstruction attacks rather than only membership inference.

Core Claim

A spurious local minimum is a local optimum that is not a global one — a trap. A strict saddle is a stationary point with at least one direction of negative curvature, which perturbed gradient descent escapes almost surely. Write the defender's problem over factors $U \in \mathbb{R}^{d \times r}$, $V \in \mathbb{R}^{k \times r}$ as

$$\min_{U,V}\; \ell_{\text{train}}\!\left(W_0 + UV^\top\right),$$

and the honest-but-curious attacker's problem over a candidate batch $x'$ with labels $y'$ as

$$\min_{x',y'}\; d\!\left(\nabla_{U,V}\,\ell(W_0 + UV^\top; x', y'),\; g\right),$$

where $g$ is the observed gradient and $d$ is squared Euclidean distance or cosine distance, the two dominant attack losses.

The core claim is that a wedge exists and can be occupied deliberately. Forward-landscape theory says training is benign when the factorization rank is generous: LoRA Training in the NTK Regime has No Spurious Local Minima proves every second-order stationary point is a global minimum once $r(r+1)/2 > KN$ (with $N$ samples and $K$ outputs), and Global Optimality in Low-rank Matrix Optimization proves the same under a restricted condition-ratio $\beta/\alpha \le 1.5$ on curvature. Yet practice runs LoRA at $r \in \{4, 8, 16\}$, far below the proven $r \approx \sqrt{N}$, and training still converges — the theory's rank floor is over-conservative. Meanwhile, on the attacker's side, shrinking $r$ compresses the signal about the input that the gradient can carry: in the linearized regime the per-sample gradient with respect to $V$ factors the input's features through a rank-$r$ projection, so distinct inputs become gradient-indistinguishable. We hypothesize, and will formalize, that the empirically sufficient training ranks are exactly the ranks where inversion becomes non-identifiable and restart-dependent — a wedge between "easy to train" and "hard to invert" that no existing work occupies.

What Prior Work Does and Does Not Show

Three strands motivate this proposal without covering it. Foreseeing Reconstruction Quality of Gradient Inversion: An Optimization Perspective is the closest: it shows reconstruction quality is predicted by extreme Hessian eigenvalues of the attack objective at the true data point (its LAVP proxy reaches a Spearman correlation of 0.51 on CIFAR-10 with $L_2$ attacks versus 0.35 for gradient norms, and 0.74 on ImageNet for cosine attacks). But it analyzes only local curvature at the ground truth — it cannot see spurious minima elsewhere on the attack surface, studies no low-rank parameterizations, and offers no defense. We extend from local curvature to global trap structure, and from prediction to design.

Second, MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning proves low rank alone is not a privacy shield — its active PEFTLeak attack, where a malicious server poisons the frozen backbone first, recovers 85.9% of image patches on CIFAR-100 (82.8% on CIFAR-10, 81% on TinyImageNet, and 72.6% even at batch size 128). Crucially, it also reports that standard passive optimization attacks fail against LoRA under normal architectures, without explaining why. That unexplained passive failure is precisely the phenomenon we aim to explain and amplify; MineGrad neither analyzes the inversion landscape nor evaluates formal DP mechanisms. Third, Landscape Learning for Neural Network Inversion shows an attacker can reshape a hostile inversion landscape by learning a mapping network from a well-behaved latent space, achieving a 10x convergence speedup on GAN inversion. This constrains our design: any trap we engineer must survive an attacker who brings their own landscape-smoothing reparameterization, so we evaluate against that attack style explicitly.

On the privacy side, Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD shows structure helps utility under differential privacy — a training procedure that adds calibrated noise to gradients, giving a formal guarantee that any single sample barely changes what is shared — reporting perplexity of 27.52 versus LoRA's 29.29 on Enron at privacy budget $\epsilon = 0.5$ with 7.6x fewer parameters. But it evaluates only membership inference, never data reconstruction, and offers no optimization theory. Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD shows worst-case guarantees cannot be favorable for both privacy and utility at once, which motivates studying instance-level hardness — what actually traps a concrete attacker — as a complement to worst-case accounting.

Research Program

The theory program has three targets. First, a non-identifiability bound: in the linearized (NTK) regime, we will prove that the set of candidate inputs inducing identical factor gradients contains a manifold whose dimension grows as the factor rank $r$ shrinks, quantifying the under-determination that Inverting Gradients - How easy is it to break privacy in federated learning? and SPEAR: Exact Gradient Inversion of Batches in Federated Learning observed empirically but never bounded for factorized updates. Second, trap existence: for batches $b \ge 2$ we will construct explicit spurious stationary points of the attack objective — mixture solutions that match the aggregate gradient without matching any individual sample — building on the entanglement analysis of Concealing Sensitive Samples against Gradient Leakage in Federated Learning, and showing the basins of these traps dominate the restart distribution at low rank. Spurious Valleys, NP-Hardness, and Tractability of Sparse Matrix Factorization with Fixed Support supplies the cautionary counterpoint: structural constraints can reintroduce traps in factorization problems, so we must verify the defender's landscape stays benign — using the strict-saddle escape machinery of Hessian-guided Perturbed Wasserstein Gradient Flows for Escaping Saddle Points to confirm perturbed gradient descent still converges — rather than assuming it. Third, a secondary question we will answer empirically: does per-sample gradient clipping and Gaussian noise, the machinery of DP-SGD, create spurious minima in the forward factorized landscape, breaking the benignness assumed by all existing theory? 1-Bit Matrix Completion under Exact Low-Rank Constraint suggests rank-deficient local minima remain globally optimal, hinting at robustness, but nobody has checked this under clipping.

The systems deliverable is a wedge search procedure: a search over a family of factorizations — matrix rank $r$, tensor-train cores with fixed support, and the strength of the balancing regularizer $\|U^\top U - V^\top V\|_F^2$ — scored by two cheap diagnostics. The forward diagnostic reuses existing benignness certificates (condition ratio $\beta/\alpha$, NTK rank condition). The attack diagnostic is new: a trap index, computed by launching many cheap random restarts of the passive inversion attack and measuring the spread and bimodality of final attack losses, plus the restart success rate (fraction of restarts exceeding a reconstruction-fidelity threshold). A factorization deep in the wedge certifies training and shows high trap index; we then validate the pick under full-strength attacks.

Proposed Experiments

Four experiments, all on benchmarks represented in the literature. First, a synthetic matrix-sensing setting in the style of Global Optimality in Low-rank Matrix Optimization (whose scaling runs go from matrix size 40 to 5120), where the inversion landscape can be mapped exactly by homotopy continuation and grid search, verifying the non-identifiability dimension bound and the location of constructed spurious minima. Second, vision: Vision Transformer with LoRA adapters on CIFAR-10, CIFAR-100, and TinyImageNet. Passive attacks are Deep Leakage from Gradients (which achieved reconstruction MSE of ~0.0069 on CIFAR-100 in its original full-model setting) and the cosine-distance-with-TV-prior attack of Inverting Gradients (which reached PSNR of ~14 dB on a trained ResNet-20 on CIFAR-10, versus ~7 dB for Euclidean matching); we also run a generative-prior variant in the style of Gradient Inversion with Generative Image Prior and a mapping-network variant in the style of Landscape Learning, since the latter is the strongest known counter to hostile landscapes. No reported result found on Lacuna for passive optimization-based inversion against LoRA updates under standard architectures, so we establish this baseline ourselves. The worst-case active reference is PEFTLeak's 85.9% patch recovery on CIFAR-100 at batch size 32. Third, language: GPT-2 (124M) with LoRA on the Enron email corpus, attacked with the language-model-prior inversion of Dimitrov et al., measuring exact token-sequence recovery, membership-inference AUC, and perplexity, against TTLoRA-DP's reported perplexity of 27.52 at $\epsilon = 0.5$ (LoRA's 29.29) and its non-private MIA AUC of 88.68% versus LoRA's 94.91%. Fourth, the frontier: for each defense — plain LoRA, DP-LoRA, TTLoRA-DP, our landscape defense alone, and ours combined with DP-SGD — we plot reconstruction fidelity and attack success against utility, at matched tuning budgets, tracing whether geometry shifts the privacy-utility Pareto frontier.

Baselines

The "do nothing" reference point is standard LoRA fine-tuning with gradients or adapter updates shared in the clear. It is the strongest leakage baseline because MineGrad's PEFTLeak already demonstrates its worst case — 85.9% patch recovery on CIFAR-100 and 82.8% on CIFAR-10 at batch size 32, degrading only to 72.6% at batch size 128 — and we will rerun the passive attacks (DLG, cosine-TV, generative-prior, mapping-network) on identical architectures, data splits, and batch sizes to get the honest-but-curious numbers that MineGrad reports only anecdotally. Beating it means cutting passive restart success rate and patch recovery substantially (we target at least a halving) while holding accuracy loss under one point.

The strongest existing defense on this question is TTLoRA-DP from Privacy Enhanced PEFT: tensor-train adapter cores with ghost clipping and DP-SGD, reporting Enron perplexity 27.52 versus LoRA's 29.29 at $\epsilon = 0.5$ with 7.6x fewer trainable parameters. It is the strongest competitor because it is the only method in this literature that already combines aggressive parameter reduction with formal privacy, and it holds the best published privacy-utility numbers on the Enron benchmark. We run it with the same base model, data, and tuning budget as our method, and additionally subject it to gradient inversion — an attack it was never evaluated against. Beating it means matching or improving perplexity at $\epsilon = 0.5$ while achieving measurably lower reconstruction success under the language-model-prior attack; no reported result found on Lacuna for TTLoRA-DP's reconstruction vulnerability or its MIA AUC under actual DP noise, so we measure both.

Two ablations isolate the proposal's key ingredient. The first is a parameter-count-matched control: an adapter with the same total number of trainable parameters but generic, non-engineered structure (a random dense projection of matched dimension). If it defends as well as our wedge-selected factorization, then dimensionality — not landscape geometry — is the mechanism, and the proposal's core claim is falsified. The second removes only the trap-inducing structure (relaxing the fixed support or raising the rank past the wedge) while keeping parameters fixed, checking that attack success rises as predicted by the theory.

Why not just…?

Why not just use DP-SGD? DP-SGD carries a formal guarantee, which landscape engineering does not, and we do not claim otherwise. But the file contains two warnings: SPEAR found its exact analytical inversion resilient to low-level DP noise, and Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD proves worst-case favorable privacy-utility pairs cannot be guaranteed. The honest position is that DP is a real threat to this proposal's practical relevance, and the frontier experiment settles it: if DP-LoRA or TTLoRA-DP dominates our defense at every utility point, the hypothesis is dead and we will report that. Our claim is the narrower one that geometry shifts the frontier, and that the combination beats either alone.

Why not just lower the LoRA rank? Rank reduction is the obvious cheap version of this idea, and MineGrad already refutes it as sufficient: PEFTLeak recovers 85.9% of patches at ranks as low as $r = 8$ by aggregating across layers and rounds, and large batches do not help either, since Inverting Gradients reconstructs recognizable images from batches of 100 and SPEAR achieves pixel-perfect recovery (PSNR above 100 dB) for batches up to 25. Our parameter-matched ablation directly tests whether rank (a scalar) or factorization structure (the geometry) does the work; if rank alone explains everything, we adopt that simpler story.

Why not just screen architectures with LAVP-style curvature metrics? LAVP is a predictor, not a design tool: it measures Hessian eigenvalues at the true data point, says nothing about spurious minima elsewhere on the attack surface, and its correlations (0.51 on CIFAR-10) leave most variance unexplained. It also cannot anticipate an attacker who reshapes their own landscape, which Landscape Learning shows is practical. We will nevertheless compute LAVP alongside our trap index; if LAVP predicts restart success as well as our global diagnostics, the simpler metric wins and we will say so.

Why not just convex relaxation for the defender? Convex training sidesteps landscape questions entirely, but Global Optimality in Low-rank Matrix Optimization shows convex methods' compute explodes as matrix size grows from 40 to 5120 while factored gradient descent scales cleanly — infeasible for modern fine-tuning. Moreover, convexity of the defender's problem says nothing about the attacker's landscape, which is the object that matters here.

Decisive Checks

Three outcomes would falsify or decisively weaken the proposal, and we design for them up front. If the parameter-matched ablation defends as well as the engineered structure, geometry is not the mechanism. If the mapping-network attacker (in the style of Landscape Learning) collapses the trap index at wedge configurations, the defense fails against a realistic adaptive adversary and only the theory survives. And if active attacks like PEFTLeak still succeed against wedge configurations — MineGrad suggests they might — the honest scope of the claim shrinks to passive honest-but-curious adversaries, which is still the standard threat model for gradient-inversion defenses and the regime where the unexplained failure of current attacks begs for exactly the analysis proposed here.