Low-rank adaptation (LoRA) fine-tunes a large language model by training two small matrices beside each frozen weight matrix instead of updating the whole network. It has a reputation for not memorizing its training data. Recent work found that LoRA almost never reproduces training text verbatim or in paraphrase, even when examples are repeated twenty times. That result is about regurgitating long passages. Benchmark contamination, where test questions leak into training data, needs far less memory to do damage. To inflate a multiple-choice score, a model only has to link a recognizable question template to one answer letter, which is a few bits per item. We propose to test the claim that LoRA's resistance to text leakage does not carry over to resistance against score inflation. We also propose a training fix that does not require knowing which items leaked.
The fix adds a representation-level term to fine-tuning. Each training question is paired with label-free "blanked" variants, copies with the answer-determining span removed so the original answer no longer follows. A contrastive loss, meaning a loss that pulls chosen pairs of internal representations together and pushes others apart, keeps the question close to its rephrasings and far from its blanked copies. A model trained this way cannot answer from surface template alone. The contrastive term competes with the task loss, and the right weight between them cannot be tuned on the quantity we care about, because in practice nobody knows which training items are contaminated. We therefore set the weight automatically with a per-layer gradient-balancing optimizer applied only to the adapter parameters. This makes the balancing cheap enough to use. It also turns hyperparameter sensitivity from a nuisance into the central design constraint.
Core Claim
The proposal makes two claims, and the second only matters if the first holds. First, under controlled injection of benchmark items into a LoRA fine-tuning set, LoRA shows a measurable inflation gap: higher accuracy on the injected items than on statistically identical held-out items. This gap persists in regimes where text-level leakage, measured as in Leaner Training, Lower Leakage, stays near zero.
Second, a contrastive consistency term reduces this gap more than data augmentation does at matched clean accuracy, provided three things hold. The negatives are blanked variants of the same question. The loss is the decoupled form that works at small batch sizes. The weight between the two losses is set by adapter-level gradient balancing. The new element is the label-free blanked negative. It lets the method reach natural-language benchmarks such as MMLU-style or WMDP items, where nobody can compute a counterfactual answer. DyVal notes that dynamic regeneration is mostly limited to formal domains, and that is the gap this element addresses.
Where This Sits
The motivating tension is between two bodies of evidence. Leaner Training, Lower Leakage measures verbatim, paraphrase, and idea-level plagiarism on Arxiver and ChatDoctor. It finds LoRA's paraphrase plagiarism near 0% under 20× duplication, against over 50% for full fine-tuning, and it treats LoRA dropout as a privacy knob. The opposing evidence comes from three papers. DyVal shows that WizardMath-13B and phi-1.5 score near zero on freshly generated reasoning graphs despite strong static scores. PaCoST compares a model's confidence on original and rephrased items with a paired t-test and flags contamination in open models. LaRA shows that after reinforcement-learning post-training, likelihood-based detectors fall to chance while hidden-state geometry still separates seen from unseen problems. None of these studies isolates LoRA, and the plagiarism study never measures benchmark accuracy. Our first experiment sits exactly between them. If its answer is "LoRA does not inflate scores," the paper becomes an extension of the plagiarism result to benchmarks, and we would report it that way.
The method borrows from three places. From Decoupled Contrastive Learning it takes the loss that removes the positive pair from the denominator. That paper shows this lets contrastive training work at batch 256 instead of 4096. We need the same property at an even smaller scale: a single-GPU LoRA run, of the kind Platypus reports (13B on one A100 in five hours), holds only a handful of questions per step. From MTAdam it takes per-layer magnitude matching against an anchor loss, plus per-term second moments. MTAdam was only tested on full networks, where a separate gradient per loss term multiplies gradient memory. Restricting it to adapter parameters shrinks that memory overhead to the adapter's size. Extra backward passes through the frozen network remain, and MTAdam reports about +20% per-iteration time for this reason, so we measure that cost rather than assume it away. From LaRA's blanked perturbations it takes the negative-construction idea. LaRA uses these variants only to audit a finished model. We turn them into a training signal.
Two adjacent works constrain the design. FairNet already combines a contrastive loss with LoRA. It aligns minority-group and majority-group features for fairness, and it uses a fixed loss weight. It shows that contrastive LoRA is trainable but says nothing about contamination or about balancing. SC-LoRA balances acquiring a new task against preserving old knowledge through a covariance-based initialization. Its authors concede that weights drift away from the chosen subspace during training. That concession is the reason we place balancing in the optimizer rather than at initialization.
Research Program
The work proceeds in three stages.
Stage one: controlled contamination study. We fine-tune Llama-2-7b on Open-Platypus, using the 25k-question set from Platypus, after injecting benchmark items at duplication rates of 1×, 5×, and 20×. The 20× rate mirrors the plagiarism study. Injected items take three forms: verbatim, paraphrased, and answer-only.
We use two benchmark sources. The first is WMDP, which PaCoST treats as a clean benchmark for models pre-trained before its May 2024 release. We split it randomly in half, inject one half, and hold out the other. The inflation gap is the accuracy difference between the two halves. Because the split is random, the halves have the same difficulty distribution and no generator is needed. The second source is DyVal. We inject generated items at difficulty levels D1–D4 and compare against freshly generated items at the same level. This gives a regenerable reference in formal domains.
We vary LoRA rank, dropout, and target modules. The target-module comparison is attention projections against feed-forward blocks, following mtLoRA's finding that the attention softmax amplifies gradient conflict. We also run full fine-tuning as a reference. Alongside the inflation gap, we replicate the plagiarism study's text-leakage metrics and run PaCoST's paired test on each trained model.
Stage two: the training objective. Let $h(x)$ be the mean-pooled hidden state at a middle layer for question $x$, computed with the adapter active. Let $x^{+}$ be an offline paraphrase of $x$. Let $\mathcal{N}(x)$ be the set of negatives, made up of $x$'s blanked variants plus the other questions in the batch. The consistency loss is
$$\mathcal{L}_{\text{con}} = -\frac{\text{sim}(h(x), h(x^{+}))}{\tau} + \log \sum_{k \in \mathcal{N}(x)} \exp\!\left(\frac{\text{sim}(h(x), h(k))}{\tau}\right),$$
where sim is cosine similarity and $\tau$ is a temperature. This is the decoupled form, so the positive pair does not appear in the denominator. Training minimizes $\mathcal{L}_{\text{task}} + \lambda_\ell \mathcal{L}_{\text{con}}$, where $\mathcal{L}_{\text{task}}$ is next-token cross-entropy and $\lambda_\ell$ is set per adapter layer $\ell$ by MTAdam. The task loss serves as MTAdam's anchor, because it is the more stable of the two. If the anchor misbehaves, we fall back to a slowly updated running average of its magnitude.
Blanked variants remove the span with the highest lexical overlap with the gold option, or a key numeric or entity span. This is where a false negative can enter: removing an irrelevant span leaves the answer intact, so the "negative" is really a positive. The small-batch contrastive literature flags this risk. We audit a sample of blanked variants by hand and also report results with the negatives filtered by a second model's judgment.
Stage three: diagnosis. We compute LaRA's stability metric on injected and held-out items before and after training. This tests whether the objective removes the geometric signature of contamination or merely moves it. If inflation falls but the signature remains, the model has hidden its contamination rather than lost it, and we would report that as a failure.
Baselines
The do-nothing reference is vanilla LoRA (rank 16, attention projections) trained on the contaminated mixture with cross-entropy only. Its inflation gap is the quantity every other method must reduce. Full fine-tuning on the same mixture brackets the high end.
The strongest existing competitor for "LoRA already prevents this" is LoRA with the plagiarism study's privacy knob of higher adapter dropout, together with lower rank. On text leakage, that paper reports paraphrase plagiarism near 0% for LoRA against over 50% for full fine-tuning at 20× duplication on GPT-2 Small. No reported result found on Lacuna for benchmark-score inflation under injected contamination for LoRA or any of its variants, so stage one produces the first such numbers. We sweep dropout and rank over the same grid size we give our method. Beating this baseline means a smaller inflation gap at equal held-out accuracy, not merely at an arbitrary operating point.
The data-hygiene competitor is a Platypus-style keyword and embedding-similarity filter run before training. It should catch verbatim injections, so the informative comparison is on paraphrased and answer-only injections.
Two augmentation baselines receive exactly the extra generated text our method sees: the same paraphrases and the same blanked variants. The first is paraphrase augmentation with cross-entropy. The second, on DyVal only, is labeled counterfactual augmentation, where perturbed graphs come with their recomputed answers.
For utility we anchor to SC-LoRA. On the MetaMathQA-to-GSM8k/MATH suite it reports an average math score of 31.25, against 23.62 for vanilla LoRA and 28.60 for full fine-tuning. We run that suite to confirm the contrastive term does not cost math ability, and we count anything clearly below vanilla LoRA's score as a failure.
Ablations isolate each ingredient. We drop the blanked negatives and keep only paraphrase positives. We replace the decoupled loss with standard InfoNCE at batch sizes 8 to 32. We replace MTAdam with a fixed $\lambda$, in two versions: one swept against the true inflation gap, which is an oracle unavailable in practice, and one swept against held-out accuracy only, which is what a practitioner could actually do.
All runs share data, base model, seeds (at least three), a single-A100 budget, and an equal number of tuning trials. MTAdam's extra wall-clock time is reported rather than hidden.
Why not just…?
Why not just deduplicate the training set against benchmarks? Filtering is the right first step, and we run it. But PaCoST finds that contamination survives rephrasing and partial answer leaks, and DyVal shows structurally memorized patterns collapse on regenerated items. Our paraphrased and answer-only injections test this directly. If the filter removes nearly all of the inflation in those conditions, the training-side method has little reason to exist, and we will say so.
Why not just lower the rank or raise LoRA dropout? Leaner Training, Lower Leakage shows these settings suppress text leakage, but text leakage is not score inflation. Answer-key memorization needs far fewer bits than regurgitating a passage. Stage one's rank-and-dropout sweep is the test. If low rank alone closes the inflation gap without hurting held-out accuracy, our first claim fails, and the result is still worth publishing.
Why not just augment with paraphrases or counterfactuals under ordinary cross-entropy? This is the most serious threat. Paraphrase augmentation of a leaked item teaches the leaked answer for the paraphrases too. That should spread the inflation and make paraphrase-based detectors blind rather than remove the problem. We measure this directly through the held-out-half gap and PaCoST p-values. Labeled counterfactual augmentation, by contrast, plausibly works as well as our method wherever counterfactual answers can be computed. That is why we restrict it to DyVal and why our claim of novelty rests on WMDP-style natural-language items, where blanked variants have no answer label to train on. If counterfactual augmentation matches our method on DyVal, that confirms rather than undermines the design. The open question is whether the label-free version transfers.
Why not just grid-search the loss weight? Grid search needs a target metric, and the target here, inflation on unknown leaked items, is unobservable in deployment. The fixed-$\lambda$ ablations make this concrete: we compare the oracle sweep, the practitioner's sweep on held-out accuracy, and MTAdam. MTAdam reports recovering 97.9% against 88.8% for Adam on MNIST under poorly chosen initial weights, but that comes from a very different setting. If the practitioner's sweep matches MTAdam, adaptive balancing is unnecessary, and we will drop it from the recommendation.
Why not just evaluate on dynamic benchmarks instead of fixing training? For formal reasoning, that is often the better answer, and we use DyVal for exactly that purpose. It does not cover multidisciplinary, knowledge-heavy benchmarks, and it does nothing to make published static scores trustworthy. Those static scores are the reason the training-side question matters.
Decisive Checks
The project lives or dies on two measurements.
The first is whether vanilla LoRA's inflation gap on the held-out WMDP half exceeds seed-to-seed variance at 5× duplication while its text-level plagiarism stays near zero. If it does not, the paper reports that LoRA's memorization resistance extends to benchmark scores, and the mitigation stage is dropped.
If the gap does appear, the second check is whether the contrastive objective with blanked negatives, balanced by MTAdam, reduces it more than paraphrase augmentation does. That comparison is made at held-out accuracy and GSM8k/MATH utility within noise of vanilla LoRA, and the reduction must be accompanied by a matching drop in LaRA's stability signature. A reduction in the gap that leaves the geometric signature intact would mean the method hides contamination rather than prevents it, and we would report it as negative.
References
- Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- Decoupled Contrastive Learning
- Platypus: Quick, Cheap, and Powerful Refinement of LLMs
- MTAdam: Automatic Balancing of Multiple Training Loss Terms
- FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRA
- SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
- Scalable Multi-Task Low-Rank Model Adaptation