Does Domain Knowledge Make Clinical Chain-of-Thought a Better Liar?

When a language model is asked to "think step by step" before answering, we hope the written reasoning is what produces the answer. A growing body of evidence says it often is not. Frequently the model settles on an answer from surface cues and then writes a plausible justification for it, which is called post-hoc rationalization. In clinical text this matters twice over. Step-by-step prompting there actually lowers accuracy for most models. And clinical benchmarks are known to contain shortcuts, meaning cues in the question that predict the label without reading the evidence. This proposal tests a specific, uncomfortable hypothesis. Further pre-training on medical text, the standard way to specialize a model (domain-specific pre-training), strengthens the model's prior beliefs about which clinical claims "sound true." Step-by-step reasoning then becomes a fluent way to rationalize those beliefs rather than a way to read the patient record or trial report.

The test bed is clinical trial inference. A model reads a trial report and decides whether a statement about it is entailed or contradicted. A model that never sees the report already scores far above chance on this task. We build two models that differ only in domain exposure. We split the test items according to whether a statement-only shortcut model gets them right or wrong. Then we measure whether the written reasoning tracks the evidence or tracks that shortcut. If the hypothesis holds, we test a targeted fix: preference training on pairs of trial reports that differ in one decisive fact. This fix rewards reasoning whose answer flips when the evidence flips. It differs from existing faithfulness training, which perturbs the reasoning text rather than the evidence.

Core Claim

The claim has two parts. The diagnostic part says the "clinical paradox" and the clinical shortcut problem are the same failure seen from two sides. The paradox is that step-by-step reasoning hurts clinical accuracy. The shortcut problem is that models answer from the statement alone. In both cases, reasoning traces in this domain draw on the model's stored medical knowledge instead of the supplied record, and domain pre-training makes this worse, not better. Why Chain of Thought Fails in Clinical Text Understanding evaluated 95 models on the BRIDGE clinical benchmark and found that 86.3% did worse with step-by-step prompting. It traced the drop to traces that bring in outside medical knowledge absent from the record and that lose exact dosages and abbreviations. NLI4CT found that a BERT model reading only the statements reaches F1 0.626, against a best full-input baseline of 0.644. Neither paper asks whether the reasoning trace is a rationalization of that statement-level prior, and neither varies domain exposure while holding everything else fixed.

The intervention part says that rewarding sensitivity to evidence counterfactuals moves models toward reasoning that actually carries the answer. Evidence counterfactuals are minimal edits to the trial report that flip the gold label. Rewarding sensitivity to reasoning-step perturbations does not do this as well. The intervention should raise accuracy specifically on items where the shortcut is wrong, without the large accuracy penalty that On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models reports for faithfulness steering.

Research Program

Matched models. Comparing an off-the-shelf biomedical model with an off-the-shelf general model confounds domain exposure with everything else. We therefore build the pair ourselves. Starting from one open 7B base model, we run continued pre-training on clinical trial and biomedical text: a language-modeling adapter in the style of the one merged by Edinburgh Clinical NLP at SemEval-2024 Task 2, plus a full-parameter variant if compute allows. The control is the same base model trained for the same number of tokens on general text. Both then receive identical instruction tuning on NLI4CT training data, with and without reasoning traces.

Measuring shortcut-driven rationalization. We train a statement-only classifier with k-fold cross-fitting on the training set, so every test item gets an out-of-fold shortcut prediction. We split the test items into a shortcut-aligned subset, where that prediction is right, and a shortcut-conflict subset, where it is confidently wrong. We then run three probes on each model's reasoning traces.

The first is evidence ablation. We replace the trial report with an unrelated report from the same disease area and record how often the answer and the overall argument stay the same. A model that reasons from evidence should change its answer. A rationalizer will produce a confident, fluent trace anyway.

The second is early answering. This is the area-over-curve measure from On the Hardness of Faithful Chain-of-Thought Reasoning: truncate the trace at successive steps and record when the final answer becomes determined.

The third is an unsupported-content rate. It counts medical entities and numbers in the trace that do not appear in the report, extending the lexical analysis used in the BRIDGE study.

The prediction is specific. The domain-adapted model should show higher statement-only accuracy when we probe its own representations. On shortcut-conflict items it should agree more often with the shortcut under step-by-step prompting. It should also have earlier answer commitment and more unsupported content. Its accuracy on shortcut-aligned items may still rise. The parallel-path finding of How Likely Do LLMs with CoT Mimic Human Reasoning?, where the prompt drives the answer and the reasoning separately, is the general-domain result we expect domain pre-training to intensify. That paper never tested domain-adapted models.

The fix: evidence-counterfactual preference training. For each training item we create a minimally edited trial report that flips the gold label. Edits change one outcome number, swap the arms of a comparison, or remove the decisive eligibility criterion, reusing the numerical and contradiction perturbation types from SemEval-2024 Task 2. We sample reasoning traces under both the original and the edited report. A trace is "preferred" if its answer is correct in both versions and every number or entity it quotes appears in the report it was given. A trace is "dispreferred" if its answer stays the same across the flip. We train on these pairs with direct preference optimization.

The method is borrowed from FRIT, which builds faithful/unfaithful pairs by inserting Wikidata or arithmetic distractors into reasoning steps and reports +7.6 points on GSM8K for Mistral-7B. The difference is where we intervene. FRIT tests whether the answer depends on the reasoning. Our failure mode is that both the reasoning and the answer ignore the evidence, and step perturbations cannot detect that. The approach also differs from CoPeD, which removes question-to-answer correlations inherited from a teacher's faulty reasoning by training the student to detect and correct errors. CoPeD never manipulates the evidence the answer should depend on.

Baselines

The naive reference points are direct zero-shot answering and statement-only prediction. The statement-only floor is F1 0.626, reported by NLI4CT. Every system must beat it clearly on the shortcut-conflict subset, where a pure shortcut model scores near zero by construction. Direct answering without reasoning matters most here, because the BRIDGE study found it beats step-by-step prompting for most models. We run it on both matched models with the same instruction tuning.

The strongest external reference points are frontier and large open models. On NLI4CT-P, GPT-4 zero-shot reached an average score of 0.8328 with task Faithfulness 0.9479, per Edinburgh Clinical NLP at SemEval-2024 Task 2. The best task F1 was 0.80, from Mixtral-8x7B-Instruct systems, per SemEval-2024 Task 2. The strongest 7B competitor is Edinburgh's merged domain-plus-triplet-loss adapter on MistralLite-7B, which reported an average score of 0.7862. We reimplement it on our base model with the same data and tuning budget. We do not expect a 7B model to beat GPT-4's average score. Success means matching the merged adapter's overall score within one point while clearly beating it on shortcut-conflict accuracy and on evidence-ablation sensitivity. Note that the benchmark's own "Faithfulness" metric is defined by correct prediction flips under contradiction perturbations. It is related to, but distinct from, trace faithfulness, so we report both.

The strongest faithfulness-training competitors are FRIT-style step-perturbation preference training and a Markovian bottleneck. In the Markovian setup, from Markovian Transformers for Informative Language Modeling, the answer is produced from the trace alone. That paper reports GSM8K rising from 19.6% to 57.1%. Each competitor gets the same base models, the same preference-pair budget, and the same number of optimizer steps as our method. Hyperparameters for every method are tuned on one shared development split.

Two ablations isolate the key ingredients. The first trains a direct classifier with no reasoning on the same evidence-counterfactual pairs, as plain data augmentation. If it matches our method on conflict-subset accuracy, the reasoning trace contributes nothing to robustness, and we will report that. The second removes the quote-verification condition from the preference criterion, to test whether grounding the trace in the report matters beyond the answer flip.

Why not just…?

Why not just skip the reasoning and answer directly? On BRIDGE, direct answering beat step-by-step prompting for 82 of 95 models, so this is a serious option. But direct answers on NLI4CT are exactly where the statement-only shortcut lives. High aggregate F1 can hide near-chance behavior on shortcut-conflict items, and a direct answer offers no trace to audit. Our experiments report direct answering on the conflict subset and under evidence ablation. If it is as evidence-sensitive as our trained reasoning model, the clinical case for reasoning traces collapses, and that is a publishable negative result.

Why not just debias against the statement-only model, for example by down-weighting examples it gets right? This is the most serious threat. Classic hypothesis-only debiasing might recover conflict-subset accuracy at low cost. We found no reported result on Lacuna for hypothesis-only debiasing on NLI4CT. We therefore include it as a baseline, applied to both the direct and the reasoning models. It uses the same statement-only model we build for the split, which also guards against a fold-leakage advantage. Reweighting targets one identified cue, and the research file notes that such fixes can leave higher-order associations intact. Evidence counterfactuals instead target the property we actually want, dependence on the report. The NLI4CT-P numerical and paraphrase perturbations then test whether either fix generalizes beyond the cue it was built against.

Why not just use a bigger general model like GPT-4? GPT-4 leads NLI4CT-P. But How Likely Do LLMs with CoT Mimic Human Reasoning? found GPT-4 reached correct answers despite wrong reasoning steps 74.4% of the time on multi-digit addition. On the Hardness of Faithful Chain-of-Thought Reasoning found GPT-4 less faithful than smaller models because its priors settle the answer early. We run our evidence-ablation and early-answering probes on GPT-4's traces through API sampling. This tells us whether scale buys evidence dependence or only accuracy.

Why not just fine-tune on rationales, or apply FRIT or the Markovian bottleneck as published? Supervised fine-tuning on rationales weakens the causal link from reasoning to answer, per the parallel-path analysis cited above. It also carries a high-resource penalty for robustness, per Does Self-Rationalization Improve Robustness to Spurious Correlations?. FRIT and the Markovian bottleneck make the answer depend on the trace, but a trace written from the model's medical prior satisfies both constraints while still ignoring the report. The head-to-head on evidence ablation settles this. If FRIT matches us there, evidence-level intervention adds nothing.

Proposed Experiments and Decisive Checks

The primary benchmark is NLI4CT with its NLI4CT-P perturbations. We report task F1, Consistency, and Faithfulness separately for the shortcut-aligned and shortcut-conflict subsets, along with the evidence-ablation answer-retention rate, early-answering area over the curve, and the unsupported-entity rate. As a secondary transfer test, we take BRIDGE extraction tasks with trace-based prompting and check whether evidence-counterfactual training reduces the length trap: the negative correlation between trace length and accuracy, and the rate of entities absent from the record. Conflict items are relatively few, so we use bootstrap confidence intervals over items and three training seeds.

Three results would be decisive. The diagnostic claim holds if the domain-adapted model, compared with its general-text twin, shows higher shortcut agreement on conflict items and higher answer retention under evidence ablation when reasoning, with intervals that exclude zero. The claim fails if domain exposure leaves these unchanged or reduces them, which would mean domain knowledge grounds rather than biases. The intervention claim holds if evidence-counterfactual training improves conflict-subset accuracy and evidence sensitivity over FRIT, Markovian, and reweighting baselines while losing no more than one point of overall F1 against the merged-adapter reimplementation. Finally, the no-reasoning augmentation ablation determines whether the reasoning trace itself earns its inference cost in clinical inference, or whether evidence counterfactuals alone do the work.