Vision-language models are now routinely given a dedicated "anti-hallucination" training stage. Their typical failure is describing objects that are not in the image because the words usually appear together in text, and this stage teaches them to stop doing it. The stage is followed by the ordinary life of a deployed model: more fine-tuning on charts, documents, medical images and new skills. Nobody has checked whether the hallucination resistance bought in the first stage survives the later ones. This note proposes to measure that decay directly. It then tests one specific way to prevent it. The hypothesis is that preference training against hallucination changes the model's internal activations along only a few directions. If so, protecting just those directions during later fine-tuning should keep the model honest about what is in the image while leaving it free to learn new tasks.
The opportunity sits between two literatures that do not talk to each other. Continual instruction tuning methods report retention through task accuracy only. Hallucination methods report gains for a single frozen checkpoint. If hallucination resistance is fragile, every continual-learning number in the multimodal literature is missing a failure mode. Every hallucination fix also has a shelf life that no one has measured.
Core Claim
We make two claims, and each can fail on its own. The first is empirical. After a model is aligned against hallucination with preference training, sequential fine-tuning on new visual domains erodes that alignment. Hallucination benchmarks drift back toward the unaligned baseline faster than ordinary task accuracy reveals. The second is mechanistic. The behavioral change produced by anti-hallucination preference training is concentrated in a low-rank subspace of middle-layer activations. Constraining only that subspace during later fine-tuning retains hallucination resistance about as well as re-running the alignment stage after every task, and costs less plasticity than constraining all activations or outputs.
What Prior Work Establishes and Leaves Open
Two papers motivate the measurement question. Investigating the Catastrophic Forgetting in Multimodal Large Language Models fine-tuned LLaVA on image classification. It observed forgetting together with a surge of fabricated labels: LLaVA-7b's MNIST accuracy fell from 56.96% to 20.45% after one epoch on CIFAR-10. However, it measured only coarse classification through a GPT-3.5 parser, and never used POPE or CHAIR. Continual-NExT likewise names "severe hallucination" as a symptom of lifelong training but reports only accuracy and forgetting rate.
Two papers supply the starting checkpoints. Antidote and HDPO use Direct Preference Optimization (DPO), which trains the model to prefer a chosen response over a rejected one. Their preference pairs contrast hallucinated answers with grounded ones. Antidote checked that general benchmarks did not drop immediately after alignment. Neither paper asks what happens after subsequent training on other data. Our measurement study starts exactly where they stop.
Three continual-learning methods are the competitors on the retention side. Continual LLaVA keeps the backbone frozen and routes small task adapters. Progressive Multimodal Alignment shows that the vision-to-language projector drifts and adds projector experts. Adapt-∞ selects replay data with an image-grounding score that filters out samples answerable without the image. All three report retention as task accuracy. None reports whether their retained models hallucinate more than before, so we treat them both as baselines and as subjects of the measurement.
SHARP supplies the method idea. It found that faithful and hallucinated hidden states are linearly separable, with separability peaking around layers 10–15. It then steered a fixed checkpoint along extracted directions. We reuse the observation that hallucination-relevant structure is low-dimensional and lives in middle layers. Our use of it differs: SHARP steers at inference on one model, while we constrain those directions during training so they do not move.
Two constraints shape the design. Replay can provably increase forgetting shows that replaying a few old samples can turn vanishing forgetting into constant forgetting when tasks share subspaces. This makes small preference-data replay a genuinely uncertain baseline rather than an obvious winner. The second constraint concerns test-time fixes. Curing Semantic Drift (DLC) relies on a frozen CLIP referee, a separate image-text matching model. DLC also assumes grounded tokens remain among the model's top candidates. Both assumptions may break after domain shift, so we test them rather than assume them.
Research Program
The first stage builds the measurement. We take LLaVA-1.5-7B and produce an aligned checkpoint by reproducing Antidote's synthetic preference pipeline, with HDPO's targeted pairs as a second alignment recipe. We then run the COAST domain-incremental stream (ChartQA → DocQA → IconQA → MedicalQA), the UCIT stream, and the MLLM-DCL stream. After every task we evaluate the full hallucination battery: POPE in its random, popular and adversarial splits; CHAIR at the sentence and instance level; CP-Bench F1; and AMBER hallucination rate. We also evaluate ordinary task accuracy and general benchmarks (MMBench, ScienceQA).
The central quantity is the fraction of the alignment gain that remains after task $t$. It is computed as (aligned-model score after $t$ minus base-model score after the same training) divided by (aligned score before any training minus base score). Comparing the aligned and unaligned models trained identically separates "alignment was lost" from "the downstream data itself changes hallucination." We also record the order of tasks, since APD shows that order dictates which early knowledge degrades.
The second stage localizes what DPO changed. We feed a fixed anchor set of a few thousand COCO images, with neutral captioning and existence prompts, through the base and aligned models. We collect hidden-state differences at each layer and take the top-$k$ principal directions as the alignment subspace $U_\ell$ at layer $\ell$. We verify the localization claim causally. If we project these directions out of the aligned model, POPE and CP-Bench should fall back to near-baseline. If instead we patch them into the base model, it should recover much of the gain. We then track how the aligned model's anchor activations move along $U_\ell$ versus orthogonal to it across the continual stream. The prediction is that hallucination decay correlates with motion inside $U_\ell$ and not with total activation drift. We also split the analysis by component, freezing the projector versus the backbone. This tests whether lost grounding is a projector problem, as PMA argues, or a backbone attention problem, as Why LVLMs Are More Prone to Hallucinations in Longer Responses suggests.
The third stage is the retention method, which we call subspace anchoring. During fine-tuning on each new task we add a penalty that holds the aligned model's anchor-image activations fixed only along the alignment subspace:
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{x \in A} \left\| U_\ell^{\top}\big(h_\ell^{\theta}(x) - h_\ell^{\theta_0}(x)\big) \right\|^2$$
In this penalty, $A$ is the anchor set of unlabeled image-prompt pairs, $h_\ell^{\theta}(x)$ is the current model's hidden state at layer $\ell$ on input $x$, and $\theta_0$ is the aligned checkpoint. $U_\ell$ is the $k$-column subspace basis from stage two, and $\lambda$ sets the penalty strength. The method needs no preference pairs after the initial alignment, no task labels, and no stored task data. It needs only anchor images and one forward pass of the frozen aligned model, which can be cached.
Baselines
The naive reference is sequential LoRA fine-tuning of the aligned checkpoint, with no retention mechanism. It tells us how much alignment is lost by default. Continual-NExT reports standard LoRA reaching about 42% average accuracy with forgetting above 20% on its stream, but no reported result was found on Lacuna for hallucination metrics after continual tuning of an aligned model. That gap is precisely what stage one fills. We also train the unaligned base model with the same recipe, which isolates alignment loss from downstream effects.
The strongest competitor on our specific question is re-alignment: re-running Antidote-style DPO after each new task. Its static effect is well documented. Antidote reports CP-Bench F1 rising from 5.7% to 78.4% on LLaVA-1.5-7B, and HDPO reports AMBER hallucination rate falling from 35.1% to 15.8%. We will give re-alignment the same preference data, the same number of DPO steps per task as the original alignment, and its own hyperparameter search of equal size to ours. Subspace anchoring beats it only if it retains at least the same fraction of the CP-Bench and AMBER gain at the end of the stream, with equal or better final task accuracy, at lower total compute. If re-alignment retains more, our method loses on the main question.
The strongest continual-learning competitors are Continual LLaVA, PMA added to DISCO, and Adapt-∞. On task accuracy, Continual LLaVA reports a COAST domain-incremental forgetting rate of 2.58% and average accuracy of 37.08%. PMA reports UCIT mean final accuracy of 73.21% with backward transfer of −4.36, and Adapt-∞ reports 0.9% forgetting on its own stream. We run each from the same aligned checkpoint on the same streams with the authors' reported settings. We then add our hallucination battery, which none of them reported. Continual LLaVA is especially important, because its frozen backbone may preserve alignment for free. Its contextual embeddings mix in prior-task adapters, however, and could import prior-task object biases.
A KL-to-aligned-model baseline distills the aligned model's output distribution on the same anchor set, matching the anchor data and penalty budget. Two ablations isolate the key ingredient. The first anchors all activations at layer $\ell$ instead of only $U_\ell$. The second anchors a random subspace of the same rank $k$. If the random subspace does as well as $U_\ell$, the localization story is wrong and the method is just generic regularization. All methods use the same base model, streams, anchor images where applicable, GPU-hour cap, and hyperparameter trials.
Why not just…?
Why not just re-run the anti-hallucination DPO after every task? This is the real threat, since the preference data already exist and Antidote reports no general-benchmark loss from alignment. Two facts keep it from settling the question in advance. Re-alignment leaves the model unprotected while it trains on each task. And nobody has shown that repeated DPO rounds interleaved with domain tuning stay stable, rather than eroding new-domain skills such as medical VQA, which Antidote's general benchmarks do not cover. We will run it at matched compute. If it matches subspace anchoring on retention and new-task accuracy, we will report the measurement and localization results as the contribution and recommend re-alignment, rather than argue for our method.
Why not just mix a small fraction of the preference pairs into each new task's training? Preference replay is cheap, but Replay can provably increase forgetting shows that small replay buffers can hurt when tasks overlap in subspace. Adapt-∞ reports that random replay reaches only 43.4% average accuracy versus 50.5% for selected data. We include preference replay at several buffer sizes to see whether its retention curve is non-monotonic in buffer size, as the theory predicts.
Why not just re-extract SHARP steering vectors after each task and steer at inference? That would fix behavior at test time without retraining. However, it needs labeled hallucination data (HaloQuest) after every update. It also adds inference cost. And it can only steer what the representation still encodes: if fine-tuning has overwritten grounded features, as the ICON ablation shows for latent drift, steering has nothing to push toward. We run re-extracted SHARP on the final checkpoints of the naive stream. If it fully recovers POPE and CP-Bench, then the alignment was hidden rather than erased, which would itself be a finding, and training-time protection would be unnecessary.
Why not just apply a decoding fix such as DLC at test time? DLC's CLIP referee is untested on charts, documents and medical images, and it cannot help with presupposition questions. We apply it to the COAST checkpoints and report CHAIR separately for in-domain COCO images and domain images. A large gap between the two would show where the referee breaks.
Decisive Checks
Three results decide the proposal. First, if the naive stream retains most of the alignment gain on POPE adversarial, CP-Bench and AMBER after four COAST tasks, the premise fails and we report a negative result: alignment is robust. Second, if projecting out $U_\ell$ does not undo the alignment, or a random subspace anchors as well as $U_\ell$, the mechanistic claim fails. Third, the method stands only if subspace anchoring retains at least as much of the CP-Bench and AMBER gain as matched-compute re-alignment, while staying within the reported forgetting levels of Continual LLaVA and PMA on task accuracy. Measuring every method on both axes, on the same streams, is the part no existing paper has done.
References
- Investigating the Catastrophic Forgetting in Multimodal Large Language Model Fine-Tuning
- Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
- Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
- Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
- Progressive Multimodal Alignment for Continual Instruction Tuning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Replay can provably increase forgetting
- Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Understanding Catastrophic Interference On the Identifibility of Latent Representations