When a medical guideline changes, a language model trained before the change still "knows" the old recommendation. The standard remedy is retrieval-augmented generation (RAG): search a document collection at question time and paste the relevant text into the prompt. Retrieval is not enough, though. Even when the model is handed exactly the right, current document, it often answers with the old recommendation it memorized in training. This proposal asks a narrow, answerable question about that failure. When a model sees a revised guideline and still gives the outdated answer, did it fail to take in the new content, or did it take it in and then let its memorized belief win? The two failures need different fixes. The first calls for better reading or better documents. The second calls for an intervention inside the model at the point where the memorized belief overrides the document.
We propose to build a small, clinician-checked benchmark of real guideline revisions and to use internal probes to sort each failure into one of these two types. A probe here means a simple classifier trained on the model's hidden activations to read out what the model represents. We then test a gated intervention that nudges the model toward the document only when the probe shows the model has encoded the current recommendation but is not using it. The main requirement is selectivity. A useful fix must accept real updates without making the model more gullible toward stale or wrong documents. Current tools that push models to follow context have never been tested on that trade-off in a specialist domain.
Core Claim
The claim has two parts. First, a large share of what When Cases Get Rare calls the "oracle gap" (models failing even when the ground-truth document is in the prompt), and what SituatedQA calls the "update fallacy" (a 15-point accuracy deficit on updated versus stable facts even with a refreshed corpus), is override rather than non-reading. On guideline-revision questions, a linear probe on mid-layer activations should recover the revised answer in many cases where the model's output gives the superseded one.
Second, the conflict detectors from prior work do not tell "the document is newer and right" apart from "the document clashes with memory." A detector that only flags a clash will therefore help on updates and hurt on stale documents. A gate that also asks whether the model encoded the revised recommendation from the document should keep most of the gain while avoiding most of the harm. If override turns out to be rare, the finding still matters: it would show that internal steering is the wrong lever for domain obsolescence and that effort belongs in reading and retrieval.
What Prior Work Establishes and Leaves Open
Three papers motivate the gap by showing that models ignore correct context. FaithEval benchmarks context faithfulness under unanswerable, inconsistent and counterfactual passages. It finds a "curse of strong priors": GPT-4o falls from about 96% to 47.5% on counterfactual contexts. Its counterfactuals are synthetic, not real updates. SituatedQA shows that about 16.5% of open-domain questions change answers over time and that refreshed passages do not close the gap. When Cases Get Rare documents "document grounding" failures and an oracle gap on rare clinical cases, but explicitly does not look inside the model. Our benchmark carries the first two into a real domain with real revisions, and our analysis supplies the internal diagnosis the third leaves open.
Two papers supply the method, and their limits are why this project is needed. CLEAR trains a small neural-network probe on final-layer hidden states to detect when context contradicts memory, then fine-tunes attention toward the conflicting tokens. It reaches 74.4% F1 on FaithEval with LLaMA-3.1-8B against 71.6% for Context-DPO. Its probe detects divergence, not which side is current. It also reports a "confirmation bias" in which models amplify irrelevant context that agrees with memory. SPARE decomposes activations with sparse autoencoders, which are dictionaries that split activations into more interpretable features. It steers a handful of features toward context-following and finds that the knowledge-selection signal peaks in middle layers. It works on single-entity swaps (NQSwap, Macnoise) and always pushes in one direction, toward context. We reuse both techniques as detectors and interventions. We differ in three ways: we test them on multi-sentence recommendation changes rather than entity swaps, we add a probe for the content of the revised answer, and we measure what happens when the context is the thing that is wrong.
Two temporal methods constrain the design. Mitigating Temporal Misalignment predicts how long a fact stays true and discounts confidence as it ages. This cut calibration error by 50–60% on SituatedQA and reached 20.5 exact match while retrieving for only 45% of queries. Question Answering under Temporal Conflict parses text into timestamped (subject, relation, object, time) records. It reaches 81–83% on Temporal Wiki against 78% for standard RAG. Both depend on explicit timestamps or start dates. Guideline revisions are often implicit, so a recommendation silently changes between editions. That is why we include an oracle condition where timestamps cannot help. Training and Evaluation of Guideline-Based Medical Reasoning trains models to execute Sepsis-3 criteria step by step, and notes that it never tested a transition such as Sepsis-2 to Sepsis-3. We take that transition as a seed case. Med-PRM verifies reasoning steps against retrieved guidelines and reaches 80.35% on MedQA with an 8B model. Its guideline store is static, so a verifier grounded in an old edition would reward the outdated answer.
Research Program
The first component is a guideline-revision benchmark. With clinicians, we will curate recommendation-level changes between successive editions of clinical guidelines, starting from the Sepsis-2 to Sepsis-3 definition change. Each item is a question whose correct answer differs between the old and new edition. We pair each item with the passage from each edition and write it in multiple-choice form, so that the old and new answers are both explicit options. Scoring is then exact, and a probe can be trained to read which option the model represents. We will also write stable-control questions from the same guidelines where the answer did not change. We do not know in advance how many revisions can be curated, so the benchmark size is an open variable. Because the analysis depends on per-item agreement, every item will be checked by at least two clinicians.
Each item is run under four context conditions. In current-oracle, only the new passage is given. In stale-oracle, only the old passage is given, so the correct behavior is to answer from it while flagging that it may be outdated, or to abstain if the question asks for current practice. In mixed, both passages are given without dates. In mixed-dated, both are given with edition years. Comparing these conditions separates reading failures, override failures and date-resolution failures.
The second component is failure attribution. For open models (LLaMA-3.1-8B as in CLEAR, and a Gemma model with public sparse-autoencoder dictionaries as SPARE requires), we train, layer by layer, a linear probe $q_\ell$. This probe reads from the activation at layer $\ell$, at the final question token, which answer option is supported by the passage in context. It is trained on stable-control items and general reading-comprehension items, never on revision items, so it cannot simply memorize the old answer. For a failed current-oracle item, if some middle layer decodes the new answer with high confidence, we count it as "encoded but overridden"; otherwise it is "not encoded."
To check that the probe reflects something the model uses, we run activation patching: we copy the middle-layer activations from a run where the model answered correctly into a failed run and test whether the output flips. We also run the CLEAR-style and SPARE-style conflict detectors on the same items. This tests directly whether they fire equally on current-oracle and stale-oracle items, which would mean they sense clash rather than currency.
The third component is the gated intervention. Let $h_\ell$ be the residual activation at layer $\ell$, $v$ the SPARE steering direction toward context, $c(h)$ the conflict detector's score, and $q(h)$ the probe's confidence that the passage supports an option different from the model's memorized one. We apply
$$h_\ell' = h_\ell + \alpha \, \mathbb{1}[c(h) > \tau_c]\, \mathbb{1}[q(h) > \tau_q]\, v,$$
where $\alpha$ is the steering strength and $\tau_c, \tau_q$ are thresholds tuned on a development split. The gate limits steering to cases where the model has clearly extracted a different answer from the document. This alone does not solve the stale-document case, because a stale passage also yields a clearly extracted answer. So we test a third gate term: a probe for the edition or date cue in context, and, where no such cue exists, a probe trained on recency statements in the general-domain SituatedQA timelines. Whether any internal signal of recency exists without explicit dates is itself one of the measured outcomes. We expect it may not, and we will report that result honestly.
Baselines
The naive reference points are closed-book answering and plain RAG with the current-oracle passage and a neutral instruction. On related data, When Cases Get Rare reports GPT-5.2 at 56% closed-book and 82.2% with top-5 retrieval on OGCaReBench, which bounds how much context alone helps in a clinical setting. No reported result was found on Lacuna for any model on guideline-revision questions, because the benchmark does not yet exist.
The strongest existing competitor for making a model follow context is CLEAR, reported at 74.4% F1 on FaithEval against 71.6% for Context-DPO (CLEAR). We will re-implement its conflict-aware fine-tuning on LLaMA-3.1-8B. It is the strongest competitor because it combines a conflict probe with training that shifts attention to the document, exactly what an update requires. To keep the comparison fair, it gets the same general-domain training pool and the same hyperparameter search budget as our probes. It gets no revision items, since our method sees none either. We will also run a variant fine-tuned on a held-out portion of revision items, which is a stronger, domain-adapted version of the competitor.
SPARE (SPARE) is the direct ablation of our key ingredient. It is identical steering with the gate removed, applied unconditionally. SPARE reports roughly 10% gains over other representation-steering methods and 7% over in-context prompting on NQSwap, but gives no clinical numbers.
For explicit time reasoning, we run the timestamped structuring pipeline from Question Answering under Temporal Conflict on the mixed-dated condition. It reports 81–83% on Temporal Wiki. A second ablation removes the currency term from the gate, to show whether that term alone accounts for any selectivity.
Beating the baselines means the following. The gated method must match or exceed unconditional SPARE and CLEAR on current-oracle update accuracy. At the same time, it must show a clearly lower rate of wrongly asserting the old passage's answer as current practice on stale-oracle items, and no loss on stable controls. A method that wins on updates by becoming more credulous does not count as a win.
Why not just…?
Why not just filter retrieved documents by date or tell the model to prefer the newest source? On mixed-dated items this is a real threat, and the timestamp pipeline baseline tests it directly. If it closes the gap there, we will say so. The core of the proposal, however, is the current-oracle condition, where only the new passage is present and there is nothing to filter. SituatedQA and OGCaReBench both report failures in exactly this kind of setting. Date filtering also cannot handle the mixed-undated condition, which reflects how many clinical corpora actually store guidance.
Why not just fine-tune the model to follow context, as CLEAR or Context-DPO do? This could plausibly match our update accuracy, and the domain-adapted CLEAR variant is included for that reason. The question is the cost. The FaithEval results and CLEAR's own confirmation-bias finding suggest that fine-tuning for context-following makes models defer to whatever is in the prompt. The stale-oracle condition measures this directly. If fine-tuned CLEAR matches us on both update accuracy and stale-document harm, the gate adds nothing, and we will report that.
Why not just verify answers with a guideline-grounded reward model like Med-PRM? Med-PRM checks each reasoning step against a fixed store of guidelines. It is therefore only as current as that store, and it inherits the judge model's own memorized beliefs whenever the store is ambiguous. It also cannot tell us whether the base model read the revision. We will run it as an outside check on current-oracle items with the current edition in its store. Mainly, though, it is a separate tool from the diagnosis this proposal is after.
Proposed Experiments
The first decisive check is the failure split. On current-oracle revision items, we measure what fraction of failures are "encoded but overridden," with the patching test as confirmation. If this fraction is small, internal steering is ruled out as the main remedy, and the program becomes a benchmark-and-reading study. If it is large, the oracle gap has an internal site, and the intervention experiments are justified.
The second check is detector selectivity. We compare the AUROC of the CLEAR-style and SPARE-style conflict detectors on current-oracle versus stale-oracle items, layer by layer. We expect them to fire equally on both, which would be direct evidence that they sense clash, not currency.
The third check is transfer. We train the probes and steering directions on NQSwap and FaithEval, then evaluate them on clinical revision items. We also compare against OGCaReBench oracle items, which isolate domain effects without any temporal change. This tells us whether the internal machinery for resolving conflicts generalizes across the domain gap.
The final check is the main comparison. We report update accuracy, stale-document harm rate, stable-control accuracy and abstention quality for every baseline and ablation, with confidence intervals obtained by resampling items. A benchmark of this size will likely be small, so wide intervals are expected. The first three checks are designed to be informative even if the final comparison is not decisive.
References
- When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
- Probing Latent Knowledge Conflict for Faithful Retrieval-Augmented Generation
- Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs
- Teaching Consensus Rules (and Exceptions) to LLMs for Trustworthy Medical Reasoning
- Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards