When a language model is fine-tuned on sensitive text with differential privacy (a guarantee that the trained model barely changes if any one training record is removed), every trainable parameter gets Gaussian noise added on every step. Small adapters therefore help. Tensor-train adapters, which write a weight update as a chain of small three-way "cores," have been shown to beat matrix LoRA under strict privacy budgets. Each joint between adjacent cores has a width, called its bond dimension or rank. Today those widths are picked by hand. Choosing them well is the central knob of the privacy-utility trade-off: a joint that is too wide mostly carries noise, and one that is too narrow starves the model of capacity.
We propose to set each bond width from the noise itself. The noise that DP-SGD injects has a known scale, so it leaves a predictable "noise floor" in the spectrum of each joint once the tensor train is put in a standard orthogonal form. Bond directions whose importance sits below that floor are almost all noise and can be cut. Cutting them lowers the dimension that future noise lands in and also removes the noise already stored in those directions. The rule reads only parameters that DP-SGD has already privatized. Rank selection is therefore post-processing and costs no extra privacy budget. Learned rank gates and private hyperparameter search do not have this property.
Core Claim
The claim is that, under DP-SGD, a good tensor-train rank can be read off the learned adapter. Periodically convert the cores to left-canonical form, a gauge-fixing step in which every core but the last is orthonormal. In that form the singular values at each bond (the Schmidt values) measure how much of the update passes through that joint, and they no longer depend on the arbitrary invertible matrices that can be inserted between cores.
Pure DP noise accumulated over training produces a bulk of Schmidt values with a predictable upper edge. The rule keeps bond direction $j$ at bond $k$ only if
$$s_{k,j} > \kappa \, \tau_k(\sigma, C, B, \eta, t),$$
where $s_{k,j}$ is the $j$-th Schmidt value at bond $k$ and $\tau_k$ is the largest Schmidt value the same training run would produce with zero signal. $\tau_k$ depends on the noise multiplier $\sigma$, clip norm $C$, batch size $B$, learning rate $\eta$ and step $t$. The multiplier $\kappa$ is fixed in advance, with $\kappa = 1$ meaning "at the noise edge."
For plain SGD on an $m \times n$ unfolding, $\tau_k$ scales like $\eta\sigma C\sqrt{t}(\sqrt{m}+\sqrt{n})/B$. The $\sqrt{t}$ comes from summing independent per-step noise, and $\sqrt{m}+\sqrt{n}$ is the known largest singular value of a Gaussian random matrix. For momentum or a floored adaptive optimizer, we calibrate $\tau_k$ with a "null run" on public or synthetic data under identical hyperparameters, so no private data is touched.
The prediction that follows is that the selected ranks shrink as $\epsilon$ tightens. A direction survives only when its signal clears a threshold that grows with $\sigma$. This is the qualitative behavior that Learning under Storage and Privacy Constraints found for lossy compression of noisy inputs: compress at a rate matched to the noise and the compressor acts as a denoiser. We apply that idea to the parameters of a private adapter, inside the training loop.
Relation to Prior Work
Three papers motivate this one by exposing the gap. Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD introduced TTLoRA under DP-SGD and adapted Ghost Clipping (a way to compute per-example gradient norms without materializing per-example gradients) to tensor-train cores. It reported better perplexity than LoRA at $\epsilon = 0.5$, but it fixes bond dimensions by hand across layers and adds noise to raw core gradients with no treatment of gauge symmetry. Applying Differential Privacy to Tensor Completion showed that perturbing gradients inside the loop beats perturbing the outputs for CP and Tucker decompositions, again with ranks fixed a priori. PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA showed that gauge symmetry in $AB^\top$ amplifies DP noise and fixed it by projecting onto a fixed-rank matrix manifold. It covers matrices at fixed rank only, and roughly doubles step time. Our canonical-form step addresses the same gauge issue for multi-core chains, but cheaply and only at truncation time; we do not claim a full tangent-space method.
Three papers supply the rank-selection ideas we either reuse or argue against. All of them run without privacy:
- Learning Low-Rank Tensor Cores with Probabilistic ℓ0-Regularized Rank Selection for Model Compression learns Bernoulli gates on core slices and itself warns that the gates are unstable from gradient variance.
- BATUDE enforces a global parameter or FLOP budget through an alternating Lagrangian.
- Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models prunes SVD triplets by importance score, with no formal DP guarantee.
The ℓ0 paper constrains our design: if gates are unstable without noise, they are unlikely to be stable with DP noise on top. We therefore make the rank decision with no learned gate at all.
Research Program
We will build a TTLoRA-DP training loop that starts every adapted layer at a generous maximum bond width. Every $T$ steps the loop does three things:
- Sweep the cores into left-canonical form using QR factorizations.
- Compute the Schmidt values at each bond.
- Truncate directions below $\kappa\tau_k$, then absorb the kept factors back into the neighboring cores. Optimizer state is truncated consistently with the cores.
The analysis has three pieces. First, we will establish privacy as post-processing: the rule reads only DP-SGD outputs, and the per-step noise multiplier is unchanged when dimensions shrink, so the standard accountant applies. Second, we will derive $\tau_k$ in closed form for SGD and validate the null-run calibration against it. Third, we will characterize how the rank selected at bond $k$ depends on $\epsilon$, and whether in-loop truncation beats truncating once at the end. That second question is the key ingredient, because in-loop truncation stops future noise from entering the discarded directions.
The threshold $\kappa$ and the truncation period $T$ are set on public data only, so no tuning touches the private set.
Baselines
The naive reference point is DP-SGD matrix LoRA at a small fixed rank. On Enron at $\epsilon = 0.5$ it reaches perplexity 29.29, as reported in Privacy Enhanced PEFT.
The strongest existing method on this question is hand-ranked TTLoRA-DP from the same paper, at 27.52 perplexity on Enron at $\epsilon = 0.5$. We rerun it on GPT-2 (124M) with the paper's ranks and with our seeds, clip norm, batch size and step count. Our method must match its parameter count at the end of training, or show a better perplexity-versus-parameter curve. Beating it means lower perplexity at equal $\epsilon$ and equal or fewer final parameters, with MIA AUC no higher.
On Math-10K, the reasoning-accuracy reference is PRISM at 0.690 average accuracy at $\epsilon = 6$, against 0.641 for AdamW DP-LoRA. On PTB we found no reported result on Lacuna for a DP TTLoRA perplexity, so we will report rerun numbers only.
We also run a DP port of the ℓ0-gate method, which is the obvious direct combination, under the same accounting. Two ablations isolate our ingredients. The first applies the same noise-floor truncation once after training. The second runs the in-loop rule without canonicalization, truncating the raw core slices, to test whether gauge fixing matters.
Why not just…?
Why not just select ranks on a public proxy dataset and then train privately with them fixed? This is the most serious threat, because it is also free in privacy terms. It fails exactly when the public and private data differ: the useful rank depends on both the private signal and $\sigma$, and a public run sees neither. We test it directly by choosing ranks on non-private public text (non-private ℓ0 gates, or our rule run without noise) and comparing against our in-loop rule on Enron across $\epsilon$. If the proxy matches us at every $\epsilon$, the in-loop contribution is not needed and we will say so.
Why not just sweep a few uniform ranks? Choosing the best of several private runs leaks privacy unless the selection itself is accounted for. TTLoRA-DP reported a fixed choice without that accounting. We will run an accounted private selection over uniform ranks as a baseline. The non-private evidence already favors non-uniform allocation: in the ℓ0 paper, uniform tensor rings reached 26x compression on LeNet-5 against 61.39x with learned ranks.
Why not just port learned gates or budget Lagrangians into DP-SGD? The gates receive noisy gradients like every other parameter, and the ℓ0 authors already flag gate instability without noise. We will measure how much the selected ranks oscillate and how often a layer collapses to rank zero, compared with our rule.
Why not just truncate once at the end with SVD? That is our first ablation. The evidence from Applying Differential Privacy to Tensor Completion, where output perturbation lost to in-loop gradient perturbation, suggests the end-of-training version will lag. If it does not, the in-loop machinery is unnecessary.
Proposed Experiments
The first decisive check is synthetic, using the 20×20×20 tensors with true rank 3 from Applying Differential Privacy to Tensor Completion. We will use the Tucker or tensor-train form so that a canonical form exists. The rule should recover rank 3 at large $\epsilon$ and drop to smaller ranks at small $\epsilon$, at roughly the signal levels the noise edge predicts. MovieLens-100K as a 3-way tensor then tests the same rule on real sparse data, measured by RMSE.
The main experiments are GPT-2 on Enron and PTB at $\epsilon \in \{0.5, 1, 3, 8\}$. We report perplexity, final parameter count, the rank profile per layer, wall-clock time (given the gap between parameter savings and real GPU speed noted in Tensor Methods for Language Models), and MIA AUC. Math-10K at $\epsilon = 6$ on a larger model then tests whether the gains extend beyond perplexity.
The idea fails if any of three things happens: the public-proxy rank baseline matches it across $\epsilon$; the end-of-training truncation matches the in-loop rule; or the selected ranks do not shrink as privacy tightens.
References
- Learning under Storage and Privacy Constraints
- Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD
- Applying Differential Privacy to Tensor Completion
- PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
- Learning Low-Rank Tensor Cores with Probabilistic ℓ0-Regularized Rank Selection for Model Compression
- BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition
- Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models
- Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability