When a language model is further trained on a specialty such as medicine, it gets better at exams in that field. It can also get worse at a skill that matters more in deployment: doing what the document in front of it says when that document disagrees with what the model memorized. Clinical benchmarks show the pattern. Specialized medical models trail generalist models when asked to follow supplied clinical guidelines (Benchmarking Clinical Decision Pathway Adherence in Large Language Models; CliMedBench). Meanwhile, the continual-training literature that produces such specialists measures forgetting only on broad benchmarks like MMLU and GSM8K (Efficient Domain Continual pretraining by Mitigating the Stability Gap). This note proposes to treat "follow the context when it contradicts memory" as a capability that can be forgotten during fine-tuning, and to protect it the way forgetting research protects math: by replaying a small amount of targeted data.
The replay data is the new part. It is a few thousand domain-specific conflict examples, such as "the attached 2024 guideline revises the threshold to X; what threshold applies?" Clinicians write the templates, an LLM fills the slots, and humans verify the items. The examples are mixed into domain training at roughly 1–2% of tokens. We will also watch the attention heads known to arbitrate between memory and context. The goal is to test whether the loss of context-following shows up there first, and whether targeted replay keeps those heads intact.
Core Claim
We make three testable claims. First, domain fine-tuning erodes faithfulness under conflict faster and more severely than it erodes general benchmarks. Second, generic replay does not stop this erosion, and neither does unconditional self-generated replay. Third, a small replay set of human-verified, domain-plausible conflict examples preserves faithfulness at no cost to domain gains. We also expect the erosion to be visible as drift in the "context heads" and "memory heads" identified by Cutting Off the Head Ends the Conflict before it is visible in outputs.
The second claim is where this work diverges from the closest prior work, and the reason is simple. Context-Free Synthetic Data (CFS) samples from the base model with no prompt and penalizes divergence from those samples. That approximates a divergence on the model's unconditional distribution. Behavior on a prompt that pairs a question with a contradicting document is a conditional behavior these samples almost never exercise. Generic web replay, as in the 5–10% rule of Balancing Synthetic Data and Replay, has the same gap. The same paper already reports that fragile skills like GSM8K need skill-specific replay. We hypothesize that context-following is another fragile skill of this kind.
Relation to Prior Work
The motivating evidence comes from three sources. The clinical benchmarks above document the specialization paradox. CliMedBench also shows the opposite failure, gullibility: accuracy falls from about 47% to about 8% when references are false. Guo et al. show a V-shaped "stability gap," a temporary dip in general ability early in continual training, but they never measure faithfulness.
The method borrows from two lines of work. On the faithfulness side, CANOE synthesizes 10k Wikidata-based counterfactual QA items and trains with a reinforcement learning variant (Dual-GRPO). RHIO masks retrieval heads to generate unfaithful negatives. TRUST-ALIGN uses preference optimization for grounded refusals. All three are one-shot post-training on a fixed model. None asks what happens to faithfulness when that model is later specialized, and none uses human-authored domain conflicts. On the data side, the human authoring loop follows the author–reviewer pattern: an LLM drafts, humans verify. Bae reports this is roughly 13 times more efficient than writing from scratch. The use of expert templates with programmatically fixed answers follows ToolQA, whose 117 templates guarantee that memory cannot answer the question.
One finding appears to contradict us. CFS reports that prompt-conditioned synthetic replay is worse than context-free replay, because it narrows the distribution and reinforces generated errors. We take this seriously in two ways. Our conflict set is small, and it is mixed alongside CFS or generic replay rather than replacing it. Its answers are fixed by the template and checked by humans, not generated by the model, which removes the error-reinforcement mechanism CFS identified.
Research Program
The base model is Llama-3-8B-Instruct, which CANOE also uses. The adaptation stream follows Guo et al.'s medical recipe: curated medical text, perplexity filtering, and general-text mixing. It is followed by medical instruction data.
Human-authored conflict set. Clinicians write roughly 200 templates for plausible overrides, including revised dosing thresholds, updated first-line recommendations, local formulary substitutions, and patient-specific contraindications stated in a chart excerpt. An LLM fills the slots. A second LLM audits the items for internal inconsistency, following the CliMedBench two-model pipeline. Clinicians then accept or fix each item. The answer is always determined by the supplied context.
The set also includes a matched share of items in which the context is irrelevant or unreliable in a detectable way, such as a document that is off-topic or self-contradictory. There the correct behavior is to rely on memory or flag the problem. This keeps us from simply training gullibility. Plausibility matters: stubbornness is strongest for high-frequency entities, and the domain's most common facts are exactly what specialization entrenches. Absurd swaps like "the capital of France is London" may not reach that regime.
Diagnostics during training. At fixed checkpoints we record faithfulness, general, and domain scores. We also measure the causal effect of the memory and context heads, found by PH3 path patching, and of RHIO's retrieval heads. This tests whether faithfulness has its own stability gap and whether head drift predicts it.
Baselines
The naive reference is domain adaptation with no faithfulness protection, using Guo et al.'s recipe as is. It also serves as our measurement of how much faithfulness specialization destroys. Guo et al. report a 76.7% medical average for Llama-3-Physician-8B across MMLU-Medical, PubMedQA, MedMCQA and MedQA. Our reproduction should land near that number for domain gains to count as intact.
The first replay baseline is 10% generic replay. It is the configuration Spiegelhalter et al. found best at 1.7B scale.
The second is CFS at matched replay tokens. It is the strongest data-oblivious forgetting method in the file. On OLMo-1B with MetaMathQA it held the general average at 28.24% versus 23.55% for standard fine-tuning, while reaching 29.34% on GSM8K (CFS).
The strongest competitor on the faithfulness question is post-hoc repair. We fine-tune the specialist, then apply CANOE's recipe using their released-style synthetic data, or ContextFocus steering (ContextFocus). CANOE reports a 22.6% average faithfulness gain on LLaMA-3-8B-Instruct with closed-book QA retained. For the medical specialist we found no reported result on Lacuna for CANOE or ContextFocus applied after domain adaptation.
Two ablations isolate the key ingredient. One replaces the human-verified set with an equal-sized automatically generated domain conflict set (LLM-written swaps, no human pass). The other replaces it with CANOE-style general Wikidata counterfactuals.
Fairness controls. Every arm sees the same domain tokens and the same total replay tokens. Arms without our set receive the same number of extra tokens from their own replay source. All arms get the same hyperparameter search budget. Post-hoc repair is charged the compute of its extra stage.
What counts as winning. Our method beats a baseline if it matches the naive run on domain accuracy (within one point) and on general benchmarks. It must also close at least half the faithfulness gap between the specialist and the base model on NQ-SWAP and ConFiQA. Finally, it must not worsen CliMedBench's false-information accuracy or SafeRAG attack rates.
Why not just…?
Why not just repair faithfulness afterward with CANOE or steering? This is the real threat. If post-hoc CANOE on the specialist matches our method at equal total compute, the contribution reduces to the measurement result. We would say so. We expect it to fall short for two reasons. Its general Wikidata counterfactuals may not reach domain-specific stubbornness. A second training stage on a specialized model can also trigger its own forgetting, which CANOE never measured on a specialist. On steering specifically, ContextFocus's authors limit their claims to short-form QA. MEGA-CDP's multi-step pathway adherence (Pathway Consistency Cost) is exactly where a single fixed steering vector should fail, and we test there.
Why not just use generic replay or CFS? They are cheap and well tested, and they may partly work, because they keep weights near the base model. The checkpoint curves settle this. If CFS alone keeps NQ-SWAP and ConFiQA within a point of the base model, targeted conflict replay is unnecessary and we would report that. Our prediction rests on the argument above: an unconditional divergence estimate does not constrain conditional conflict behavior. Spiegelhalter et al.'s observation that math needs its own replay is precedent for this kind of gap.
Why not just generate the conflict data automatically? Human verification is the expensive ingredient, so the automated-set ablation is its test. If automated domain swaps match human-verified ones on faithfulness and on false-information robustness, the human loop should be dropped or reduced to spot checks. We expect humans to matter mainly for two things: the plausibility of overrides, and the correctness of the "context is unreliable" items. Errors in those items would teach gullibility.
Why not freeze or anchor the context heads directly? If head drift predicts faithfulness loss, penalizing changes to those heads is a cheaper intervention. We include it as a third arm. If it matches data replay, that is a useful mechanistic finding in its own right. PH3's cross-domain robustness of the heads makes the anchoring arm plausible. Coarse head-level interventions may still miss diffuse long-form conflicts.
Proposed Experiments
Faithfulness. We evaluate on ConFiQA's three subsets and NQ-SWAP, reporting the rate of answering from context versus from memory. We also use GroundBench for long-form faithfulness. RHIO-13B's 83.77% FaithScore there is a reference point, not a target. For clinical use of context we use MEGA-CDP's Pathway Consistency Cost and accuracy.
Guards against gullibility and poisoning. We use CliMedBench's false-information test and SafeRAG's White DoS and Soft Ad tasks. The latter matter because training to follow context could raise the refusal attack rate, already reported as 100% for some models.
Domain and general ability. Domain ability uses the four medical benchmarks. General ability uses MMLU, GSM8K and BBH, as in CFS.
Decisive checks. The first is whether the naive specialist loses substantially more on NQ-SWAP and ConFiQA than on MMLU. If it does not, the premise fails. The second is whether CFS and generic replay leave that loss in place. The third is whether our 1–2% set removes it while post-hoc CANOE at equal compute does not. The fourth is whether the automated-set ablation falls measurably short on false-information robustness. A secondary check is whether drop in context-head causal effect precedes faithfulness loss across checkpoints. That would make head drift an early warning signal for continual training.
References
- Benchmarking Clinical Decision Pathway Adherence in Large Language Models
- CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios
- Efficient Domain Continual pretraining by Mitigating the Stability Gap
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- Context-Free Synthetic Data Mitigates Forgetting
- Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
- Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning
- Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization
- Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
- Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
- ToolQA: A Dataset for LLM Question Answering with External Tools
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models