Where Low-Resource Fine-Tuning Writes Hallucinations

When a large language model is fine-tuned on a few hundred examples, it does not only learn the task. It also learns what to do with questions it cannot answer, and the habit it picks up is often to answer them anyway. This proposal argues that this habit depends on where in the network the fine-tuning update lands. In a transformer, the attention heads decide which earlier tokens to read from. The feed-forward (MLP) blocks behave like a lookup table that maps subjects to stored facts. Our hypothesis is that, with very little data, updates to the MLP blocks write new and weakly supported subject-to-answer links. Those links then surface as confident fabrications on unrelated questions and overwrite facts the model already knew. Updates restricted to a sparse set of attention heads can change the output format and the task behavior while leaving the stored facts mostly alone.

The claim matters for two reasons. The cheapest way to make a model useful in a niche domain is a small fine-tune, and the low-resource literature has so far judged such fine-tunes almost entirely on task accuracy. The hallucination literature, for its part, has studied fine-tuning mostly at full data scale or after the fact. We propose a controlled factorial study that crosses three factors: where the update is placed, whether the base model already knew each training example's answer, and how many examples there are. We measure hallucination directly. If the locus effect holds, it gives a concrete design rule for small-data adaptation. If it fails, it rules out a popular intuition about parameter-efficient methods being "safe" for factuality.

Core Claim

At 50 to 1,000 training examples, fine-tuning has two effects we can measure. The first is damage: the fraction of held-out questions the base model answered correctly that it now gets wrong. The second is the fabrication rate: the fraction of questions the base model could not answer that now receive a confident wrong answer instead of a hedge or an inconsistent spread of guesses. We predict both are driven mainly by training examples whose answers the base model did not already know. We further predict that this effect is concentrated in methods that update MLP weights.

The proposal puts these predictions to a direct test. Sparse attention-head editing, plain attention-only low-rank updates, and soft prompts should all reach comparable task accuracy with lower damage and fabrication than MLP or full low-rank updates. The claim is falsifiable at the level of a single interaction: update location crossed with whether the base model knew the answer.

Grounding in Prior Work

Two results motivate the locus hypothesis.

The first is Joint Localization and Activation Editing for Low-Resource Fine-Tuning (JoLA). JoLA learns gated scaling and offset vectors on a sparse set of attention heads. The gates are on-off switches that a sparsity penalty pushes toward "off." At 200 examples JoLA beats LoRA, and the authors report that intervening in MLPs or full hidden states overfits badly. JoLA, however, never measured hallucination, calibration, or truthfulness.

The second is Fine-Tuning Dynamics of In-Context Factual Recall in Transformers. It analyzes a one-layer transformer whose frozen MLP stores factual triples, and shows that a handful of examples suffices to learn the attention pattern that retrieves them. Its setting is toy-scale. Its message, that attention learns how to read while the MLP holds what is known, is exactly the separation we want to test in real models.

A third result supplies the puzzle we hope to explain. Small Updates, Big Doubts applies LoRA, DoRA, and PiSSA, three low-rank update methods, to TriviaQA, NQ-Open, and SQuAD. It finds that these methods raise hallucination-detection AUROC for answer-only (black-box) detectors. AUROC here is the area under the curve measuring how well a detector separates correct from wrong answers. The same methods, however, make truthfulness less linearly decodable from the hidden states. That study used full-size training sets and varied the method family, not the update location. We ask whether the loss of linear separability is an MLP-write effect that disappears under attention-only updates.

Two results constrain the design.

Towards Objective Fine-tuning (CogCalib) finds the opposite of the simple "unknown data causes hallucination" story. Training on known data inflates confidence and calibration error, and the paper reports a 57% average reduction in expected calibration error (ECE) for Llama3-8B. ECE measures the gap between a model's stated confidence and its actual accuracy. CogCalib reports this on multiple-choice benchmarks using full fine-tuning. We therefore cannot assume that only unknown examples matter. The factorial design includes known-only and unknown-only training mixes so this disagreement can be resolved empirically in the low-resource, free-form regime.

Impact of Fine-Tuning Methods on Memorization in Large Language Models shows that prompt-based tuning keeps membership-inference AUC near 0.50, while LoRA and head tuning memorize more with each epoch. Membership inference tests whether an attacker can tell a training example from an unseen one, so 0.50 means no detectable memorization. This makes prompt tuning a necessary comparison point: it is the extreme case of "do not touch the stored facts."

Knowledge Overshadowing Causes Amalgamated Hallucination supplies a specific failure mode. Here the model blends a dominant fact with a rarer constraint. We expect fine-tuning to inject this failure when newly written associations bleed into related entities, and we measure that bleed directly.

Research Program

Base models and tasks. We use LLaMA-3-8B, the backbone JoLA was evaluated on, plus Qwen2.5-3B and Mistral-7B from Small Updates, Big Doubts. The task is closed-book question answering on TriviaQA and NQ-Open. We use SQuAD v1 as a context-grounded control, where answers are in the passage and stored knowledge should matter less.

Labeling what the base model knows. Before any training, we label every question as known or unknown to the base model. We sample ten few-shot answers per question and mark it known if the majority match the gold answer. As a second definition, we use CogCalib's negative-log-likelihood threshold, which scores how probable the model finds the gold answer. We report results under both definitions so the conclusions do not hinge on one.

Training mixes. From the labeled pool we draw training sets of 50, 200, and 1,000 examples in three mixes: all known, all unknown, and the natural ratio.

Update locations. We train each mix under five placements, with the trainable parameter budget matched as closely as the methods allow. The first is JoLA's sparse gated attention-head edits. The second is LoRA restricted to the attention projections. The third is LoRA restricted to the MLP blocks. The fourth is LoRA on all layers. The fifth is soft prompt tuning.

Measurements. Every trained model is evaluated on disjoint held-out questions, again split by the base model's knowledge. We record five quantities:

  • accuracy on held-out unknown questions;
  • damage on held-out known questions;
  • fabrication, measured as wrong answers given with high sampling consistency;
  • ECE;
  • detector AUROC, using the semantic-consistency, perplexity, and entropy detectors from Small Updates, Big Doubts, together with a linear truthfulness probe on hidden states.

Bleed probe. For each unknown training fact, we build related probe questions about the same subject with a different relation, and about a similar subject with the same relation. We then count how often the trained answer leaks into the response. This makes amalgamation-style hallucination measurable.

Hard-subset check. We add TruthfulQA and the HaluEval QA split to check whether any locus effect carries over to adversarial and recognition settings.

Mechanistic follow-up. If MLP updates turn out to be the culprit, we test whether the effect follows the location of the edit or the size of the edit. To do so, we take an all-layer LoRA, trace which layers' updates carry the training answers, and zero out only the MLP part of the update at inference time.

Baselines

The naive reference point is the frozen base model given the same examples as in-context demonstrations, with no weight change. By construction it has zero damage. It anchors how much accuracy fine-tuning buys and at what factuality cost. For context, From RAG to Reality reports a 43.4% development-set error rate for few-shot Qwen3-8B on SciHal claim classification. We include ICL nonetheless because it is the honest zero point.

The strongest low-resource competitor is JoLA. At 200 examples on LLaMA-3-8B, the JoLA paper reports 70.55% average reasoning accuracy, against 66.58% for LoRA and 56.19% for LoFIT, an earlier activation-editing method. We will rerun JoLA and LoRA with the public recipes on our QA data, giving every method the same examples, the same number of epochs, a small learning-rate sweep of equal size, and checkpoint selection on the same held-out development split. For hallucination specifically at this data scale, we found no reported result on Lacuna for any of these methods. Beating LoRA therefore means matching its held-out accuracy within one point while cutting both damage and fabrication by a margin that exceeds the seed-to-seed spread across five seeds.

The strongest factuality-aware competitor is CogCalib. It is a training-loss change rather than a change of update location: it applies label smoothing or an entropy penalty to examples the model already knows. We will apply CogCalib to all-layer LoRA at matched data and compute. If CogCalib applied to all-layer LoRA reaches the same damage and fabrication levels as attention-only updating, then location is not the decisive variable and we will report that. We will also test whether the two combine.

Finally, the MLP-only LoRA and attention-only LoRA arms are the ablation that isolates the key ingredient. They separate "where the update goes" from JoLA's other features, its gating and its tiny parameter count. If attention-only LoRA behaves like JoLA, the claim concerns location. If only JoLA succeeds, the claim narrows to extreme sparsity, which is a weaker result.

Why not just…?

Why not just early-stop on validation accuracy? The memorization result in Impact of Fine-Tuning Methods on Memorization shows that memorization rises steadily across epochs, and CogCalib shows that confidence keeps inflating after accuracy has plateaued. A checkpoint chosen for accuracy can therefore already be overconfident. We test this directly by logging damage and fabrication at every epoch for every method. If the best-accuracy LoRA checkpoint already has damage as low as attention-only updating, early stopping is a sufficient fix and the location claim is moot. We treat this as a live possibility.

Why not just drop the unknown training examples? This is the most serious threat. If fabrication comes from unknown examples, filtering them out may fix LoRA without changing where it updates. Two things push back. CogCalib reports that known-only training harms calibration, and at 200 examples filtering discards a large share of an already tiny dataset. The known-only arm of the factorial settles the question. If known-only all-layer LoRA matches attention-only updates on damage and fabrication at equal accuracy, the contribution reduces to a data-selection rule, and we would say so.

Why not just use prompt tuning, which does not memorize? Prompt tuning is included as an arm precisely because of its near-0.50 membership-inference AUC. Its known weakness is lower task accuracy than weight-updating methods on small data. If it matches attention-head editing on both accuracy and factuality, the practical recommendation becomes "use prompts." That would still support the core idea that stored facts should be left untouched.

Decisive Checks

The central test is an interaction effect between update location and whether the base model knew the answer. MLP-touching methods trained on unknown-heavy mixes should show markedly higher damage, fabrication, and bleed than attention-only methods at matched held-out accuracy. That gap should shrink in the SQuAD control, where answers come from the passage rather than from stored facts.

A second check addresses the Small Updates, Big Doubts puzzle. The drop in linear-probe separability should appear for MLP and all-layer LoRA and not for attention-only updates.

A third check is the inference-time removal experiment. Removing only the MLP portion of a trained all-layer LoRA should recover most of the lost known facts while keeping most of the task-format gains.

If none of these three effects appears at 200 examples on at least two of the three backbones, the locus hypothesis is wrong for real models. The study then still delivers the first matched hallucination measurements for low-resource adaptation methods, which no paper in this file reports.