Teaching Clinical Reasoners to Ignore the Domain's Own Background Words

Every specialized field has words that appear almost everywhere. In clinical trial reports these are patient, trial, arm, cohort, and adverse. They behave like stopwords such as the or is: they are frequent and say almost nothing about which document or answer is the right one. Retrieval researchers already know this and strip such words from queries. Language models are usually handed the full text, and there is evidence they lean on surface phrasing rather than on the facts. On NLI4CT, a benchmark that asks whether a clinical trial report supports or contradicts a statement, a BERT model shown only the statement, with no report, reached 0.626 F1. The best full models reached 0.644 (NLI4CT). Most of the apparent "reasoning" is therefore guessing from wording.

This proposal turns domain stopwords from something to delete into something to be invariant to. We estimate, from the corpus itself, which tokens are domain background and which carry the specific content of each document. We then fine-tune a 7B open model so that its prediction does not change when background tokens are perturbed. All words stay in the input at test time. The bet is that this targeted invariance reduces shortcut use and improves the consistency of clinical inference more than three alternatives: deleting the words outright, perturbing random tokens, or fine-tuning without any such constraint.

Core Claim

Treating domain-background vocabulary as a nuisance variable during fine-tuning should do two things. It should improve the robustness of expert-domain inference, measured by NLI4CT's faithfulness and consistency scores. It should also make the model more sensitive to the content tokens that should matter, such as numbers, drug names and outcome terms. The new element is the choice of what to be invariant to: a background set estimated statistically from the domain corpus, not a hand-written list and not uniform noise.

This also sidesteps a problem the stopword literature already reports, which we call the contextuality problem. A word like trial can be noise in one query and essential in another (LiHiSTO). Hard filtering for BM25, a standard keyword-matching retrieval method, caused zero-match failures on NLI4CT evidence retrieval. Those failures needed a fallback "closure strategy" that raised recall from about 0.15 to 0.64 at the cost of precision (ITTC at SemEval 2023). Because our model keeps the words and learns to discount them, the information is still available whenever it does matter.

How Prior Work Positions This

Two papers supply the method. The background-versus-content split comes from dTM-SAGE (Exploring Differential Topic Models). That work models each document group's word distribution as a shared background log-frequency plus a sparse group-specific deviation. It used the split to pick discriminative sentences for comparative summarization of scientific papers. It never touched neural reasoning, and we reuse only its decomposition, at the level of individual reports. The training scaffold comes from the Edinburgh SemEval-2024 system (Edinburgh Clinical NLP; code at semeval_nli4ct). That system merges a language-modeling LoRA adapter and a triplet-loss LoRA adapter on MistralLite-7B. LoRA is low-rank fine-tuning that trains a small add-on to a frozen model. The Edinburgh system feeds full, unfiltered reports, and its authors observed that chain-of-thought prompting lowered F1, which is consistent with distraction by irrelevant text.

Three papers motivate the proposal. The hypothesis-only result in NLI4CT shows that the shortcut problem is real. GraphRAG-Bench shows a related "right answer, wrong reason" pattern: RAPTOR reached 73.58% accuracy but only 45.53 rationale accuracy, and added context sometimes hurt multiple-choice performance. On a different benchmark, AppealCase reports that all 20 tested models stay below 50% F1 on predicting whether a first-instance judgment is reversed. The authors read this as models handling legal vocabulary on the surface without weighing the underlying rules.

The closest cross-direction system is K-COMP. It trains a 2B compressor that adds entity definitions and cuts "chatter" from retrieved medical passages. It reports that generic compressors like LLMLingua drop rare jargon. K-COMP deletes text with a learned black box and was evaluated on open-ended medical QA, not on inference where a dropped qualifier flips the label. We delete nothing, and the "noise" is defined explicitly and checkably. tRAG addresses a neighboring failure, "seen term bias" in synthetic query generation. It operates at the retrieval stage and is not a competitor here.

Research Program

The first step estimates the background set. We fit a sparse additive model over the NLI4CT trial reports, split by section (eligibility, intervention, results, adverse events). The log-probability of word $w$ in report section $d$ is

$$\log p(w \mid d) = m_w + \eta_{d,w} - \log Z_d,$$

where $m_w$ is the shared background log-frequency, $\eta_{d,w}$ is a sparse deviation under a Laplace prior, and $Z_d$ normalizes. A token is labelled background in a given report if its word has high $m_w$ and $\eta_{d,w}$ is near zero. It is labelled content if $\eta_{d,w}$ is large. The same word can therefore be background in one report and content in another, which is the contextual behavior a static list cannot express. Numbers and negations are always treated as content. Background labels are projected onto subword tokens so that a fragmented term inherits its word's label.

The second step trains with background invariance. Starting from the Edinburgh merged-adapter recipe, we add a consistency term:

$$\mathcal{L} = \mathcal{L}_{\text{task}}(x, y) + \lambda \, \mathrm{KL}\big(p_\theta(y \mid x) \,\|\, p_\theta(y \mid \tilde{x}_B)\big).$$

Here $x$ is the statement and report, $y$ the entailment label, $p_\theta$ the model's predictive distribution, $\lambda$ a weight, and $\tilde{x}_B$ a copy of $x$ with a random subset of background tokens masked, deleted or replaced by another background token. The model sees the original text at test time.

The third step builds a diagnostic used for analysis, separate from the leaderboard metrics. Background sensitivity is the mean change in label probability under background perturbation, which should be low. Content sensitivity is the change under perturbation of an equal number of content tokens, which should be high. A model that reasons over the facts should show a large gap between the two. We compute this for every system, including GPT-4-style prompting where access allows. This lets us ask whether the 0.626 hypothesis-only phenomenon shows up as high background sensitivity.

To test whether the effect transfers beyond medicine, we repeat steps one and two on AppealCase first-instance reversal prediction. There the background is legal boilerplate around court, plaintiff and judgment, while statutory provisions and amounts should surface as content.

Baselines

The naive reference point is MistralLite-7B with the language-modeling-only LoRA adapter on full reports, as described in Edinburgh Clinical NLP. We also report the statement-only floor of 0.626 F1 from NLI4CT. Any method whose gains vanish relative to a statement-only version of itself has not learned to read the report.

The strongest open competitor is the Edinburgh merged LM-plus-triplet adapter. It reports an average score of 0.7862 on SemEval-2024 Task 2, with +0.0346 F1 and +0.152 consistency over the LM-only adapter. We rerun it from the released code with the same base model, data, seeds and adapter rank. Our variant adds only the consistency term, and both get the same hyperparameter search budget. GPT-4 zero-shot, reported at an average of 0.8328 and faithfulness of 0.9479, is a ceiling for reference and not a target. Beating the strong baseline means a higher average score with consistency and faithfulness both improved, and F1 not lower, across at least three seeds.

The hard-filtering baseline is an ITTC-style approach (ITTC at SemEval 2023). We delete the background set from the input at both training and test time, once with the hand list (patient, trial) and once with the statistically estimated set. This is the closest existing use of domain stopwords on this benchmark. Its reported numbers concern evidence retrieval (test F1 0.619), not inference, so we rerun it as an inference baseline.

Two ablations isolate what is new. The first applies the same consistency loss to random tokens at the same perturbation rate, which separates "invariance to background" from generic token-dropout regularization. The second uses a frequency-only (inverse document frequency) background set instead of the background-plus-deviation split, which tests whether the per-report decomposition matters.

Why not just…?

Why not just delete the domain stopwords before fine-tuning? This is the most direct use of the stopword literature, and it may work. The documented risks are the contextuality problem and the precision collapse forced by fallback strategies in ITTC. In inference, deleting arm or cohort can also erase which group a number refers to. The hard-filtering baseline settles this directly. If it matches the invariance model on consistency and F1, the soft approach is unnecessary and we will say so. The background diagnostic would still be a contribution.

Why not just use a learned compressor like K-COMP? K-COMP shows large gains on medical QA. Its compressor is trained to keep question-aligned entities, and in NLI4CT the decisive token is often a numeric qualifier or a negation that does not look question-aligned. Its own motivating finding, that LLMLingua drops rare jargon, shows how deletion-based compression can fail silently. We do not have K-COMP's NLI4CT numbers, and no reported result was found on Lacuna for K-COMP on NLI4CT. Running it as a front end to the merged adapter is a fair optional comparison if the compressor is available.

Why not just add noise or dropout uniformly? Generic input perturbation is a real threat because it regularizes cheaply. The random-token ablation exists to answer this question. If random-token invariance matches background invariance, the claim that domain background is the specific culprit is false.

Why not just scale up or prompt a frontier model? GPT-4 does well zero-shot (0.8328 average). The question here is how a model that can be deployed locally can approach that level without memorizing shortcuts. Improving Low-resource QA shows that ChatGPT scored 20.8 EM against 58.9 for a targeted Bert-base on PolicyQA, so scale alone is not reliable in niche domains. We also measure GPT-4's background and content sensitivity. If GPT-4 shows a large gap between them, that is evidence that scale produces this property, and our method would be a cheaper route to it rather than a requirement.

Decisive Checks

Three outcomes would make the proposal succeed. The invariance model must beat the rerun Edinburgh merged adapter on consistency and faithfulness without losing F1. Its gain must exceed the random-token ablation. Its content-minus-background sensitivity gap must widen relative to every baseline. We also check that its statement-only variant still scores near 0.626 while its full-input score rises, which shows that the improvement comes from reading the report and not from better hypothesis guessing. A secondary success is reproducing the direction of these effects on AppealCase reversal F1, which sits below 50% for all models in AppealCase. Two outcomes would refute the idea: hard deletion matching the invariance model, or random invariance matching background invariance.