When Private Fine-Tuning Makes Models Make Things Up

Fine-tuning a language model under differential privacy (DP), a guarantee that no single training record can noticeably change what the model learns, is the standard way to adapt models to hospital notes or corporate email without memorizing them. The cost is usually reported as a few points of accuracy, the "utility gap." For generative models that answer factual questions, we argue this cost has a specific and predictable form. Facts that appear in only a few private records cannot rise above the privacy noise. When the model is asked about such a fact, it does not say "I don't know." It confidently substitutes a common fact it absorbed during pre-training, such as a common drug in place of the rare one actually prescribed. In other words, the privacy utility gap shows up as a particular kind of hallucination.

The proposal has two parts. The first is a controlled measurement. We test whether DP fine-tuning turns rare private facts into confident, frequency-biased substitutions, and whether the cutoff below which a fact is lost follows a simple law: a fact repeated in $k$ records survives only when $k\varepsilon$ is large, where $\varepsilon$ is the privacy budget. The second part uses that law. Because the training noise level is known, we can predict which questions the private model cannot reliably answer. A cheap DP count of how often each entity appears in the private data, checked against the predicted cutoff, then tells the model when to abstain. If this external, noise-aware signal beats the model's own confidence at flagging these errors, we have a practical way to turn part of the utility gap into honest abstention instead of fabrication.

Core Claim

Our hypothesis concerns errors on rare facts in private fine-tuning data. We expect these errors to be mostly confident substitutions toward high-frequency alternatives, not uncertain spreads of probability. The frequency below which a fact is lost should scale roughly as $1/\varepsilon$. The reasoning is as follows. In DP stochastic gradient descent, each record's gradient is clipped to norm $c$ and Gaussian noise of scale $\sigma c$ is added at every step. A fact carried by $k$ records therefore contributes signal of about $kqTc$ over $T$ steps at sampling rate $q$, while the noise along that direction grows as $\sqrt{T}\sigma c$. That gives a signal-to-noise ratio of

$$\mathrm{SNR}(k) \approx \frac{k\, q \sqrt{T}}{\sigma},$$

and standard accounting sets $\sigma \propto q\sqrt{T}/\varepsilon$ at fixed $\varepsilon$, so $\mathrm{SNR}(k) \propto k\varepsilon$. This is the learning-side mirror of group privacy: a fact held by $k$ records is protected at level roughly $k\varepsilon$. When $k\varepsilon$ is small, the fact is supposed to be unlearnable, and we predict the model fills the gap with its pre-training prior.

The derivation assumes the fact's gradients line up and that clipping binds. Real fine-tuning breaks both assumptions. Whether the law holds empirically is therefore the first thing we test.

Grounding in Prior Work

Three findings motivate the proposal. Chasing Your Long Tails showed that DP training in clinical prediction erases the influence of rare subpopulations. It reports a 22–26% AUROC drop in mortality prediction, with influence flipping toward the majority class. However, it studies classifiers, not generators that can fabricate. Jingwei Ni et al. and David Kartchner et al. show that hallucination arises when relevant knowledge is sparse and the model falls back on prior statistical associations. TruthfulQA shows those priors can be confidently wrong. We join these lines: DP manufactures exactly the sparse-knowledge condition under which hallucination appears.

The private fine-tuning methods we build on are Differentially Private Fine-tuning of Language Models, which trains low-rank adapters under DP, and PRISM, which removes the noise blow-up caused by perturbing the two adapter factors separately. Neither measures factuality. We use both as training backbones, so that any substitution effect we find cannot be blamed on a weak optimizer.

The work closest to ours operates at inference time. PEARL finds that DP decoding over retrieved private text causes hallucination, and it repairs this with an entropy-gap pipeline and a GPT-4o refiller. Privacy-Aware Decoding adds calibrated noise to logits. Differentially Private Steering privatizes activation steering vectors. All three leave the weights untouched. None asks what DP fine-tuning does to the facts stored in the weights, which is our question.

SRM-LoRA reduces hallucination by reshaping adapter update geometry. It is non-private, and in our setting it serves as a competing remedy rather than a foundation.

Research Program

The first component is a controlled private corpus. We generate fictional entities, such as patients with invented drug–condition pairs and companies with invented contracts, so the base model has no prior knowledge of them. We plant each fact in exactly $k$ records, with $k$ ranging from 1 to several hundred, mixed into realistic filler text. Each fictional fact is paired with a plausible high-frequency "decoy" that the base model would produce, so we can classify every error as one of three types. A substitution means the decoy or another frequent entity appears. A fabrication means an unrelated invention. An abstention means a hedge or refusal. We fine-tune Llama-3.1-8B and Gemma with DP-LoRA and PRISM over $\varepsilon \in \{1, 3, 6, 8, \infty\}$ and fit the empirical curve of accuracy against $k$. The law predicts that the curves collapse onto a single curve when plotted against $k\varepsilon$, and that errors below the cutoff are mostly substitutions made with high confidence.

The second component checks realism on HealthCareMagic-100k, a corpus of patient–doctor dialogues. We extract drug–condition and symptom–diagnosis pairs with a public medical entity list, bucket them by corpus frequency, and build question-answer probes whose answers come from the corpus itself. This tests whether the synthetic law transfers to natural long-tailed clinical text.

The third component is the gate. Before fine-tuning, we release a DP histogram of counts over a public entity vocabulary, so only counts are private and the key set is not. This spends a small budget $\varepsilon_h$, composed with the training budget and reported as one total. At inference, entities in the question are looked up. If the noisy count falls below the cutoff $k^*(\varepsilon)$, the model abstains, or defers to public retrieval if a public source exists. The cutoff is fit on synthetic data, so it costs no privacy. The gate adds one dictionary lookup per query, unlike PEARL's reported ~1.4 s refinement pass.

Baselines

The naive reference points are the non-private LoRA model at $\varepsilon=\infty$, which gives the ceiling, and the base model with no fine-tuning, which shows pure prior behavior and the decoy rate.

The strongest private backbones are DP-LoRA and PRISM. They are run on identical data, rank, step count, clipping-threshold sweep, and per-method tuning budget. PRISM's reported strength is 0.690 average accuracy on Math-10K at $\varepsilon=6$ versus 0.641 for AdamW DP-LoRA (PRISM). DP-LoRA reports 87.8% MNLI accuracy at $\varepsilon=6.7$ versus 90.2% non-private (Differentially Private Fine-tuning of Language Models). No reported result was found on Lacuna for tail-entity hallucination or substitution rates under DP fine-tuning, so our measurements establish the first numbers. The gate does not compete with the backbones. It sits on top of each one, and success means it improves selective accuracy on top of the better backbone.

The strongest competitor for the gate is internal confidence abstention. This means thresholding the answer's token entropy or top-1/top-2 logit margin, the signal Privacy-Aware Decoding relies on. We also include a cross-examination consistency check in the style of Agrawal et al.. All abstention methods are compared on the same risk-coverage curve, with thresholds tuned on the same held-out synthetic split. The gate beats them if it yields higher area under the selective-accuracy curve on the tail buckets at matched coverage, with the total privacy budget including $\varepsilon_h$ held equal.

Two ablations isolate the key ingredient. An oracle gate uses exact counts, which bounds what the DP histogram costs. A gate with a fitted rather than theory-derived threshold tests whether the $k\varepsilon$ law itself adds anything.

Why not just…?

Why not just abstain when the model is unsure? This is the real threat. If DP-trained models become diffusely uncertain on facts they failed to learn, entropy abstention is free and would match the gate. Our hypothesis is the opposite: the pre-training prior supplies a confident substitute. That would be consistent with TruthfulQA's confidently wrong imitation and with Agrawal et al.'s observation that false outputs carry high confidence. The decoy design settles this directly. If substitution errors carry entropy as high as true unknowns, we drop the gate and report the measurement and the scaling law as the contribution.

Why not just spend more privacy budget or warm-start on public data? Larger $\varepsilon$ and the public jumpstart of Pre-training Differentially Private Models with Limited Public Data both shift the cutoff. Neither removes it: under the law, some $k$ is always too small. These are the axes on which we sweep, and the gate's value is that it marks wherever the cutoff currently sits.

Why not just retrieve from the private records instead of memorizing them? Privacy Implications of Retrieval-Based Language Models found that a retrieval-based model leaked 54 PII segments versus 28 for a parametric model, and that DP-training its encoders hurt accuracy sharply. Private retrieval reintroduces the leakage DP was meant to prevent. Our gate defers only to public retrieval.

Why not just add hallucination-aware geometry such as SRM-LoRA? Its mask can be computed from public contrastive pairs without spending privacy. This is a fair alternative, and we run it under DP as an extra arm. However, it reshapes how updates move. It cannot create signal for a fact whose $k\varepsilon$ is below the noise floor, so we expect it to help mid-frequency facts and leave the tail unchanged.

Decisive Checks

The proposal stands or falls on four results. First, accuracy curves at different $\varepsilon$ must collapse when plotted against $k\varepsilon$ on the synthetic corpus. Second, below the cutoff, substitutions must outnumber abstentions and carry low entropy. Third, the HealthCareMagic frequency buckets must show the same ordering. Fourth, the gate must beat entropy and consistency abstention in selective accuracy at equal total budget.

As side checks, we confirm that HaluEval-QA hallucination rate, where SRM-LoRA reports 10.04% versus 11.46% for plain LoRA on Llama-3.1-8B (SRM-LoRA), and MMLU are not worsened by gating. We also confirm, with canary membership-inference audits following Differentially Private Steering, that releasing the histogram does not measurably increase leakage. If the curves do not collapse, the clean theory fails. The measured error typology would still be new, but the gate would have to fall back on an empirically fitted threshold.