Gauge-Fixed Differential Privacy for Tensor-Train Models

A tensor-train model stores a large weight array as a chain of small three-way blocks called cores. It is one of the most compact ways to train models on naturally multi-way data such as 3D scans, and it has recently been shown to improve the trade-off between privacy and accuracy when a model is trained with differential privacy. Differential privacy (DP) is a guarantee that the trained model reveals little about any single training example. It is usually enforced by DP-SGD: each example's gradient is clipped to a maximum length $C$, and Gaussian noise is added.

This note points out a hidden flaw in how DP-SGD is applied to tensor-train models. The same weight array can be written with many different sets of cores. You can multiply one core by a matrix and the next core by that matrix's inverse without changing the model at all. The length of the gradient with respect to the cores, however, does change under such a swap. So the clipping threshold and the noise scale have no fixed meaning: they depend on an arbitrary internal scaling that drifts during training. We propose to remove this ambiguity. Before each step, the cores are put into a standard (canonical) form. Clipping and noise are then applied to the per-example change in the function, measured in the tangent space of the set of fixed-rank tensor trains, rather than to raw core gradients. The target application is private learning on natively multi-way data such as 3D medical volumes, which current tensor-DP work has not touched.

Core Claim

Write the weight tensor as $W = G_1 G_2 \cdots G_d$, where each $G_k$ is a small three-way core and adjacent cores are contracted along a shared rank index. For any invertible matrix $A$, the replacement

$$G_k \to G_k A,\quad G_{k+1} \to A^{-1} G_{k+1}$$

leaves $W$ and the loss unchanged. It does change the gradient with respect to $G_k$, which becomes the old gradient multiplied by $A^{-\top}$. Global clipping on the concatenated core gradients, as in Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD, is therefore calibrated to an arbitrary choice of $A$. Isotropic noise added in core coordinates likewise spreads unevenly in function space.

We claim three things. First, DP training lets the cores drift into unbalanced scalings, and this drift silently changes both how often gradients get clipped and how large the noise is. Second, fixing the representation removes this source of utility loss. Concretely, we orthogonalize the cores into a canonical form using QR sweeps, then clip each example's tangent-space gradient $P_T(\nabla_W \ell_i)$ to satisfy $\|P_T(\nabla_W \ell_i)\|_F \le C$, add noise $\mathcal{N}(0,\sigma^2C^2 I)$ inside that tangent space, and map the update back to a tensor train by rank truncation. Third, the gain grows with tensor order, so it is largest on 3D and 4D data.

The privacy accounting is unchanged from standard DP-SGD. Canonicalization and rank truncation depend only on the current parameters, which are already private, so they count as post-processing. The per-example norm in canonical form splits into a sum of orthogonal per-core pieces. This lets us reuse the cached-contraction idea from TTLoRA-DP to compute exact per-example norms without ever forming the full gradient tensor.

Relation to Prior Work

The closest work is TTLoRA-DP (Kunwar et al.). It extends "ghost clipping" (computing per-example gradient norms without materializing per-example gradients) to tensor-train adapters on GPT-2, and it reports better perplexity and lower membership-inference risk than matrix LoRA. It supplies our efficiency technique and our strongest baseline. It clips a single global Euclidean norm of the core gradients and evaluates only text. We change what is clipped and noised, and we move to native 3D data.

Two older papers apply DP to tensor factorization itself rather than to supervised deep models. Applying Differential Privacy to Tensor Completion finds that perturbing gradients beats perturbing inputs or outputs for CP and Tucker completion. Online and Differentially-Private Tensor Decomposition shows that the private tensor power method can have a privacy cost independent of dimension when the signal is spread out (incoherent). They motivate the claim that where the noise lives matters more than raw parameter count. Neither handles tensor trains or their gauge freedom.

Two papers supply the method on the tensor side. The alternating optimizer of the Support Tensor Train Machine already relies on mixed-canonical forms to stop numerical oscillation. We read that as evidence that gauge matters for optimization, and we extend the concern to calibrating privacy noise. Remurs and DuSK (He et al.) show that keeping 3D structure helps when samples are scarce.

Two results constrain the design. The worst-case lower bound of Bu et al. caps what any DP-SGD variant can achieve, with a noise floor that falls only like $1/\sqrt{\ln M}$ in the number of updates $M$. We therefore aim to recover utility lost to miscalibration, not to beat that bound. Adaptive clipping (AdaptDP-FedAvg) tunes a scalar radius over time. It is complementary to our method, but it cannot fix a threshold whose meaning changes with the gauge.

Research Program

The first step is diagnosis. We instrument TTLoRA-DP-style training and log three quantities over training: the per-core norm ratios, the fraction of examples clipped, and the ratio of the noise norm projected into function space to the noise norm in core coordinates. The prediction is that noise acts like a random walk that inflates core norms unevenly. The imbalance should grow over training, and both the clipped fraction and the effective noise in function space should drift with it.

The second step is the method: canonical-form clipping in the tangent space, plus tangent-space noise, plus retraction by rank truncation, with exact per-example norms computed by cached contractions. The third step is a small analysis. We show that the sensitivity bound is the same for every gauge. We also show that the noise dimension equals the dimension of the fixed-rank manifold, which is smaller than the raw core count because gauge directions carry no signal. This gives a concrete explanation for why raw parameter counts overstate how much noise the model actually absorbs.

Baselines

The naive reference points are non-private training of each model, which gives an upper bound on accuracy, and ordinary DP-SGD on a flattened, unfactored model. Bu et al. document the size of the loss from the latter: roughly 80% dropping to roughly 40% for CIFAR-10 ResNet-18 at the theoretical noise bound. A second naive point is DP matrix LoRA, which represents the "subspace reduction is enough" view.

The strongest existing method is TTLoRA-DP with global Euclidean ghost clipping. On Enron at ε = 0.5 it reports a perplexity of 27.52, against 29.29 for LoRA (Kunwar et al.). We will first reproduce this number. Beating it means lower perplexity at the same ε, rank and step count, averaged over seeds, with a gap larger than the seed-to-seed spread.

On MedMNIST v2 3D we found no DP result for any method on Lacuna; the only reference there is non-private ResNet-50 3D at AUC 0.863 (MedMNIST v2). There we will run every baseline ourselves.

Two ablations isolate the key ingredient. The first is Euclidean clipping with periodic core rebalancing, which equalizes core norms without doing tangent projection. The second is per-core clipping with a separate threshold for each core in the arbitrary gauge. Fairness rules apply to all methods. They use the same data splits, the same ranks, and the same Rényi DP accountant. They share one hyperparameter grid over $C$, learning rate and batch size, of identical size, and we report the privacy cost of tuning in the same way for every method. Compute is matched in wall-clock time, and we report the overhead of canonicalization separately.

Why not just…?

Why not just use matrix LoRA under DP-SGD? On text, Kunwar et al. already show a gap in favor of tensor trains in both perplexity and membership-inference AUC. On 3D volumes, matrix adapters force the data to be flattened, which throws away the structure that Remurs and DuSK show matters at small sample sizes. We include it as a baseline rather than assuming it loses.

Why not just rebalance the core norms every few steps and keep Euclidean clipping? This is the real threat. Balancing only fixes a diagonal scaling of the gauge, so it cannot make clipping fully gauge-invariant or put the noise inside the tangent space. If it nonetheless matches our method, the correct conclusion is that gauge drift is the whole story and the cheap fix suffices. That would still be a useful finding for practitioners. The rebalancing ablation settles this directly, and we will report it whichever way it comes out.

Why not just clip each core separately? Per-core thresholds add hyperparameters but remain tied to the arbitrary gauge. The per-core ablation, run under the gauge-scaling test below, will show whether it inherits the same sensitivity to internal scaling.

Why not just use adaptive clipping? A scalar radius that adapts over time (Fukami et al.) follows the overall gradient scale but not the gauge. We will also test adaptive clipping stacked on top of our method, to check that the two are complementary.

Proposed Experiments

The first testbed is Enron and Penn Treebank with GPT-2 tensor-train adapters, at ε in {0.5, 1, 3, 8}. This anchors us to the published TTLoRA-DP number. The second testbed is the MedMNIST v2 3D tasks, in two variants: a tensor-train classifier trained from scratch on 28×28×28 volumes in the style of STTM, and tensor-train adapters on a 3D CNN pretrained on a disjoint split treated as public. The metric is AUC across the same ε values. ADHD-200 (200 subjects) serves as a small-sample stress test. We expect near-useless privacy at that size and report it honestly. We also vary tensor order by reshaping the same weights into 3, 4 or 6 cores, to test the prediction that gains grow with order.

The decisive check is a gauge-scaling test. At initialization we apply an adversarial but function-preserving rescaling of the cores by a factor $c \in \{0.1, 1, 10\}$. If our diagnosis is right, final utility under Euclidean TTLoRA-DP will change substantially with $c$, while under gauge-fixed clipping it will not change beyond seed noise. If Euclidean training turns out to be insensitive to $c$ and the logged core imbalance never grows, the premise is wrong and the proposal fails cleanly.

As a secondary check, we run DLG gradient inversion (Zhu et al.) against a single noised step at matched ε. This measures whether moving the noise into function space changes reconstruction error in either direction.