Pruning Memorization Out of a LoRA Adapter, One Singular Direction at a Time

When a language model is fine-tuned on private data, it can end up able to recite individual training records word for word, such as a patient's dialogue or an injected secret string. Low-rank adaptation (LoRA) is the dominant cheap way to fine-tune. It freezes the original weights and learns a small additive update $\Delta W = BA$ of rank $r$. That update is a finished, compact object that can be inspected and edited before an adapter is shipped. We propose to split each layer's update into its singular directions, the independent rank-one pieces it is built from. We then score each piece by how much it helps the model on its training data versus on held-out data, and shrink the pieces that help only on training data. The hope is to remove verbatim memorization while keeping the task skill the adapter was trained for. The method does not need to know which records were memorized.

The opportunity comes from one striking result. Cheap Forgetting fine-tuned Llama-3.2-1B on Pile text seeded with 100 canaries (synthetic 16-digit secrets, each repeated 30 times). It then scaled the whole adapter by a single scalar $\alpha$. Canary extraction stayed near 97% down to $\alpha \approx 0.6$ and then collapsed to 0% at $\alpha = 0.5$. At matched general-text likelihood, the scaled adapter leaked 7% of canaries against 71% for early stopping. A single global knob throws away every direction of the update by the same factor, including the ones carrying useful adaptation. The paper names this as open: it applies "a uniform scalar across all adapter layers and singular directions." Our claim is that memorization is not spread evenly across the adapter's singular directions. It concentrates in a small, identifiable subset. Removing that subset gives a strictly better trade-off between leakage and retained task skill than uniform scaling, lower-rank training, or early stopping.

Core Claim and Mechanism

For each adapted weight matrix, write the learned update in its singular value decomposition, $\Delta W = \sum_{i=1}^{r} \sigma_i u_i v_i^\top$. Here $\sigma_i$ are the singular values and $u_i, v_i$ the corresponding output and input directions. We attach a gate $\gamma_i \in [0,1]$ to each term, so the edited update is $\sum_i \gamma_i \sigma_i u_i v_i^\top$. Each term's contribution to overfitting is estimated by a first-order score:

$$ s_i = \frac{\partial}{\partial \gamma_i}\Big[\mathcal{L}_{\text{train}}(\gamma) - \mathcal{L}_{\text{held}}(\gamma)\Big]\Big|_{\gamma = 1} $$

Here $\mathcal{L}_{\text{train}}$ is the loss on a sample of the fine-tuning data and $\mathcal{L}_{\text{held}}$ is the loss on a held-out split from the same distribution. A strongly negative $s_i$ means the direction lowers training loss much more than held-out loss, so it carries information specific to the training examples. One backward pass per data split gives all scores at once. We then either threshold the scores or run a few steps of optimizing the gates. The objective penalizes the train–held-out gap and keeps held-out loss near its unedited value.

This differs from nearby work in three ways. Plain spectral truncation keeps the largest $\sigma_i$. Spectral Thresholding in Continual LoRA uses that idea to allocate rank across tasks, not to remove private content. Our ordering comes from the generalization gap, not from magnitude. LoRA unlearning methods need an explicit set of records to forget: LoKU uses the forget set's Fisher information to initialize adapters, and VILA uses adapter gradients to estimate parameter importance. Both need a forget set and, for VILA, a paired retain set, and both re-optimize. Our edit is post-hoc, retrain-free, and needs no forget set. It targets memorization introduced during fine-tuning, which is the practical pre-release situation where the developer does not know which records leaked. Finally, Memory Adapters and Memorization Sinks isolate memorization by changing how training is done. Our method applies to adapters that were already trained the ordinary way.

The hypothesis could simply be false. The claim in LoKU that LoRA's subspace is "too restrictive" to forget without re-optimization suggests one failure mode: memorized content may be entangled with task content in the same few directions. In that case per-direction gating would do no better than uniform $\alpha$. We treat this as the central empirical question, not an assumption.

Research Program

The first stage is a localization study. We train adapters in two regimes and ask whether memorization is concentrated in a few singular directions. The first regime reproduces the Cheap Forgetting canary setup. The second uses natural-language fine-tuning on ChatDoctor-HealthCareMagic and Arxiver, following Leaner Training, Lower Leakage, with injected i2b2-style canaries as in the federated LoRA study so that ground truth exists. For each adapter we remove directions one group at a time in three orders: by our score, by ascending singular value, and at random. We record canary extraction and held-out task loss after each removal. If memorization is concentrated, extraction should fall steeply under score order while held-out loss barely moves, and the other two orders should not show that pattern.

The second stage turns the score into an editing procedure and maps its Pareto frontier. The procedure has two variants, thresholding and short gate optimization, and it is applied per layer. On the frontier, the x-axis is the fraction of fine-tuning gain retained, defined as (base held-out loss minus edited held-out loss) divided by (base held-out loss minus unedited held-out loss), together with ROUGE-L for the dialogue task. The y-axis is leakage. We measure the retained gain rather than general Pile likelihood because $\alpha$-scaling can look "free" when its NLL returns to the base model level, even though that return means the adaptation has been undone.

The third stage tests robustness in the ways the unlearning literature says matter. We report leakage under stochastic sampling using leak@$k$ up to $k = 128$, because Leak@$k$ showed that greedy-decoding evaluations hide near-total leakage from methods like NPO. We report paraphrase-level leakage with the all-MPNet-Base-V2 cosine-above-0.8 criterion from Leaner Training. We report per-record leakage on the most distinctive records, since Rare, Distinctive, Memorized and Underestimated Privacy Risks for Minority Populations show that average-case metrics hide outliers. We run a relearning probe in the spirit of the Janus Interface: a small LoRA fine-tune on 10 canary pairs, testing whether the other canaries resurface. Every experiment is repeated with adapters placed on attention projections and on feed-forward blocks. Earlier leakage studies placed adapters on attention only, whereas unlearning work locates stored facts in feed-forward layers. We expect localization to differ between the two placements.

Baselines

The naive reference is the unedited adapter at $\alpha = 1$. In the Cheap Forgetting setup it has a reported canary extraction rate of 97%. Every method is compared against this leakage ceiling and against its full task gain.

The strongest existing post-hoc competitor is uniform linear interpolation from Cheap Forgetting, swept over $\alpha \in [0,1]$ in fine steps. The paper reports 0% extraction at $\alpha = 0.5$, and 7% extraction at matched NLL of about 2.25 against 71% for early stopping. It is the strongest competitor because it is also post-hoc, forget-set-free, and nearly free to compute. It uses the same adapters, and we give it the same held-out split to choose its operating point. We beat it if, at every leakage level, including leak@128 and paraphrase leakage, our edit keeps a larger fraction of task gain, with the gap exceeding seed-to-seed variation across at least three training seeds.

Early stopping uses the same run's checkpoints and is selected on the same held-out split. It represents the "just train less" answer.

Two ablations isolate our key ingredient, the gap-based ordering. The first is magnitude-ordered SVD truncation of the same adapter at the same number of kept directions. If it matches our method, the score adds nothing beyond spectral shape. The second is a fresh adapter trained at the reduced rank $k$ that our edit ends up using, with the same data, steps, and learning-rate search. It tests whether we are only rediscovering "use a smaller rank." We also include LoRA dropout at the rates studied in Leaner Training as a training-time regularizer with the same tuning budget.

DP-LoRA is the formal-guarantee comparison. Rare, Distinctive, Memorized reports 90% leakage reduction at $\epsilon = 1$ on medical vision models. For text generation on these datasets we found no reported result on Lacuna for DP-LoRA, so we will run it at $\epsilon \in \{1, 8\}$ under matched compute.

The oracle is an adapter retrained with the canaries removed. It is the upper bound on what any data-agnostic edit could achieve. For a targeted comparison in which the forget set is known, we also run NPO and VILA-initialized unlearning on the adapter. On this canary setup we found no reported result on Lacuna for NPO or VILA, so these numbers will be produced here.

Why not just…?

Why not just scale the whole adapter down, as Cheap Forgetting does? This is the most serious threat. If memorized and task-relevant content share the same directions, uniform scaling is already optimal and our gates collapse to a constant. The localization study settles this directly. If score-ordered removal does not drop extraction faster than random removal at equal held-out loss, the claim fails and we will report it as a negative result. The reason to expect otherwise is that the $\alpha$-cliff sits exactly where general NLL returns to the base model. That coincidence suggests uniform scaling achieves safety by undoing adaptation, which is the cost our retained-gain metric is designed to expose.

Why not just train a lower-rank adapter, or add LoRA dropout? Leaner Training found that higher rank slightly raises leakage and that dropout suppresses it, so both are plausible substitutes. However, they must be decided before training, while our edit applies to adapters already trained, shared, or merged. The matched-rank retraining control tests whether the edit only recovers what smaller rank would give. If a rank-$k$ adapter matches the edited rank-$r$ adapter on both axes, we will recommend the simpler recipe.

Why not just retrain the adapter without the sensitive data? LoRA retraining is cheap, so when the leaked records are known this is the right answer, and we include it as the oracle. The point of our method is that developers usually do not know which records the adapter memorized. Retraining requires an audit such as the differential training in Rare, Distinctive, Memorized, which the authors describe as prohibitive per record. Our edit needs only a held-out split.

Why not just use DP-LoRA? Differential privacy gives a guarantee our method does not. Rare, Distinctive, Memorized shows it can preserve accuracy in vision models. On the other hand, the backdoor extraction study reports that DP fails against engineered extraction without large utility loss, and DP must be adopted at training time. We will report where each method sits on the same frontier. We present the edit as a cheap post-hoc complement, not a replacement for a guarantee.

Decisive Checks

Three results would decide the project. First, in the canary setting, removing the top-scored directions must cut extraction to near zero while retaining clearly more task gain than $\alpha$-scaling at the same extraction rate. This must hold for both attention and feed-forward placements, or it must fail informatively for one of them. Second, the advantage must survive leak@128 sampling and the relearning probe. If leakage returns under either, the edit only suppresses rather than removes, the same failure Leak@$k$ documents for optimization-based unlearning. Third, on natural medical dialogue, the edited adapter's paraphrase leakage on the most distinctive records must approach the retrained oracle while keeping ROUGE-L within noise of the unedited adapter. If magnitude-ordered truncation or matched-rank retraining does as well on all three checks, we will conclude that the gap score is unnecessary.