Verbalizer Brittleness as a Symptom of Distraction in Clinical Entailment Classifiers

Entailment-based classifiers decide whether a document supports a sentence that names a candidate label, such as "the drug inhibits this gene." The document is called the premise and the label sentence the hypothesis. These classifiers are widely used in biomedicine because they need few labeled examples. Two weaknesses are well documented but have only been studied separately. First, predictions change when the hypothesis is reworded, which we call verbalizer brittleness. Second, accuracy drops when the relevant sentence is buried in irrelevant text, which we call contextual distraction. This proposal argues that, for entailment classifiers reading long technical documents, these are largely one failure. Rewording the hypothesis changes which irrelevant premise sentence the model latches onto through surface word overlap. If that is true, the disagreement among a few rewordings can act as a cheap test that tells a deployed classifier when it is being distracted and should abstain.

The question matters because published clinical and biomedical entailment results come almost entirely from clean, pre-cut premises. Clinical trial NLI systems were scored on short section extracts, and biomedical relation extraction was scored on single sentences with gold entity mentions. Real use instead means whole trial reports and full abstracts, where most entity pairs have no relation at all. The proposal measures how robust domain-pretrained encoders, general NLI encoders, rerankers, and prompted LLMs are under controlled, realistic noise. It then tests whether hypothesis-rewording disagreement predicts distraction-induced errors better than the model's own confidence does.

Core Claim

The claim has two parts, and each can be falsified on its own. The first part is a mechanism. In entailment classifiers, the instability of predictions across meaning-preserving hypothesis rewordings is small on clean premises. It grows with the number of distractor sentences that share vocabulary or entities with the hypothesis. The flipped predictions can be traced to the distractor sentence that best overlaps each rewording. The second part is a use. Because brittleness tracks distraction, the rate at which predictions flip across about four rewordings is a better signal for selective abstention under noise than maximum softmax probability. The second part follows only if the first part holds.

The mechanism is grounded in prior findings. Issues with Entailment-based Zero-shot Text Classification shows that NLI-tuned classifiers lean on lexical overlap rather than implication. It also shows that raw BERT next-sentence prediction can match them, reaching 50.6% on Yahoo topics and 73.4% on Snips. That paper did not vary the premise. Our claim is that the overlap reliance it found has a specific consequence when the premise contains many candidate sentences. A different wording overlaps a different distractor, so the wording appears to matter even though the underlying cause is the noise.

The second part of the claim builds on On the Relation between Sensitivity and Accuracy in In-Context Learning. That work (SENSEL) used prompt-perturbation instability as an abstention signal for decoder in-context learning. It gained +4.1 area under the F1-coverage curve on GPT-J-6B. It never tested entailment cross-encoders, specialized domains, or premise noise. We transfer the idea and give it a reason to work in this setting.

Research Program

Noise protocol. The first component is a controlled noise protocol built on two existing benchmarks. On NLI4CT (BpHigh at SemEval-2023 Task 7; Stanford MLab at SemEval 2023 Task 7), each statement is judged against one clinical trial report section. The natural distractors are the other sections of the same report. We insert them at increasing amounts, from zero up to the full report. On ChemProt and DDI, as used by Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?, we expand the single-sentence premise. Distractors are drawn from the same abstract or from other abstracts that mention the same entities.

Distractor types. We separate three kinds of distractor so that word overlap can be varied on purpose. Random in-domain sentences have low overlap with the hypothesis. Same-document sentences have moderate overlap. Same-entity sentences that state a different relation or outcome have high overlap. Labels never change, because the original evidence sentence stays in place. This mirrors the ability-versus-knowledge distinction in Breaking Focus: Contextual Distraction Curse in Large Language Models: the model sees the answer and simply fails to use it.

Rewording sets. The second component is a set of meaning-preserving hypothesis rewordings. For relation extraction, these are alternative label templates in the style of NBR and Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction. For NLI4CT, where the hypothesis is a free clinical statement, they are paraphrases, which annotators will check on a sample to confirm meaning is preserved. Each instance gets four rewordings. We measure the flip rate across rewordings as a function of distractor count and type.

Tracing flips to distractors. For each flipped prediction, we delete premise sentences one at a time. This identifies which sentence moves the entailment score most under each rewording. We then test whether that sentence is a distractor that shares more words with the rewording that flipped than with the other rewordings.

Model families. The analysis covers four families under identical inputs. The first is a general NLI cross-encoder, DeBERTa-xxLarge-MNLI as in the TACRED work, plus bart-large-mnli. The second is domain-pretrained encoders: PubMedBERT-large tuned on NLI4CT as in BpHigh, and BioLinkBERT inside NBR. The third is a cross-attention reranker, Qwen3-Reranker-8B, the top system in BTZSC. The fourth is a prompted open instruction-tuned LLM. This lets us ask whether domain pretraining flattens the slope of brittleness against noise. One reason to expect it might is that domain vocabulary gives models something other than surface overlap to rely on. The answer could also go the other way: the domain-pretraining literature summarized in the file documents tokenizer fragmentation and transfer gaps between formal and informal text (BioBART).

Baselines

Naive reference. The do-nothing reference is each classifier scored once, with its default hypothesis wording, on the noisy premise. Its abstention rule is maximum entailment probability. We report the drop from the clean to the noisy premise as the basic robustness number. No reported result found on Lacuna for any entailment classifier on full-report or distractor-augmented NLI4CT, ChemProt, or DDI premises. Measuring that drop is therefore itself a contribution.

Strongest existing clean-premise system. The strongest published NLI4CT systems are the fine-tuned PubMedBERT-large cross-encoder from BpHigh (test macro F1 0.690) and few-shot GPT-3.5 with retrieved demonstrations (macro F1 0.694, same paper). Stanford MLab's encoder ensemble reached F1 0.662. We retrain PubMedBERT-large with BpHigh's unfrozen recipe on the same training split. On clean premises it must reproduce roughly 0.69 before any noisy comparison counts. For relation extraction, the reference is NBR, which reports gains of 1.10 to 1.79 F1 over BioLinkBERT with full data and +34.25 F1 in 8-shot ChemProt over standard supervised models. The file gives no absolute NBR scores, so we re-run NBR ourselves on matched splits. NBR is the strongest competitor for the abstention question because it already contains an abstention detector trained with margin-based regularization.

Retrieve-then-classify. This is the most serious practical baseline. A sentence selector first filters the noisy premise, and the cross-encoder then classifies only the selected evidence. We run BM25, which Stanford MLab used and reported evidence-retrieval F1 of 0.461. We also run a dense biomedical retriever, and an oracle that keeps only gold evidence sentences. The oracle bounds how much of the noise problem retrieval can solve.

Noise-augmented fine-tuning. We retrain each encoder on training premises that already contain the distractor types. This uses the same data and number of epochs, and the same hyperparameter search budget as the clean models. It is the cross-encoder analogue of the preference training on clean versus distracted pairs in Breaking Focus.

Abstention baselines. We compare against maximum softmax probability, predictive entropy, and NBR's abstention detector. The proposed signal is rewording disagreement. To isolate what drives it, we include an ablation that perturbs something other than the hypothesis wording. It uses four random premise-sentence orderings at the same number of forward passes. This ablation separates "rewording" from "any perturbation at matched cost."

What counts as winning. All methods receive the same compute in forward passes per label. Beating the baselines means rewording disagreement achieves a higher F1-coverage area than both maximum probability and the matched-cost ordering ablation under heavy noise, across at least two of the three model families. A margin comparable to SENSEL's +4.1 would be convincing.

Why not just…?

Why not just retrieve the evidence sentence first and classify that? This is a real threat to any claim of a better classifier, and the proposal is built to absorb it. Stanford MLab's BM25 selection reached only 0.461 evidence F1, so naive retrieval passes much of the noise downstream. A strong dense retriever might do far better. If oracle and dense retrieval remove both the accuracy drop and the rewording flips, we will report that retrieval quality is the bottleneck. In that case the diagnostic still stands: brittleness was distraction, and practitioners should fix retrieval rather than tune templates. The abstention signal stays useful in either case, because retrieval cannot signal its own misses. We therefore evaluate disagreement on top of the retrieval pipeline too.

Why not just fine-tune the encoder on distracted premises? Augmentation may well close the accuracy gap in the supervised setting, and we treat it as a co-equal baseline rather than a straw man. It does not address the low-resource and zero-shot regimes, where NBR shows entailment framing matters most. There, the eight-example training sets leave little room to cover distractor variety. The experiment checks whether augmented models still show a slope of flips against overlapping distractors. If they do not, the mechanism claim holds only for unaugmented models, and we will say so.

Why not just ensemble the rewordings or threshold confidence? Averaging rewordings at the same cost as our signal is included as a prediction method. The comparison asks whether averaging improves accuracy as much as disagreement improves abstention. Confidence thresholding is the default we must beat. Prior evidence suggests it is weak under distraction: Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents reports 0.98 confidence on distracted wrong answers. Whether entailment heads behave the same way is exactly what is tested.

Why not just prompt a frontier LLM, or tell it to ignore irrelevant text? Compartmentalised Agentic Reasoning for Clinical NLI reports 23.1% mean macro accuracy for direct prompting on its 200-instance clinical benchmark. Breaking Focus reports that "ignore irrelevant info" instructions were ineffective. We include one prompted LLM with and without that instruction, so the claim is checked directly on our noise protocol rather than assumed.

Proposed Experiments and Decisive Checks

Main experiment. The main experiment crosses four model families, three distractor types, and five noise levels on NLI4CT, ChemProt, and DDI. We report macro F1, per-instance flip rate across rewordings, and F1-coverage area for each abstention signal. For DDI, abstention is the bulk of the task, since up to 85% of pairs are unrelated. We therefore also report precision on the no-relation class.

Check one: does brittleness grow with noise? The mechanism claim fails if flip rate is about as high on clean premises as on noisy ones. It also fails if flips do not grow faster with high-overlap distractors than with random ones.

Check two: are flips caused by distractors? The claim also fails if leave-one-out tracing assigns flips to the gold evidence sentence rather than to distractors. That outcome would mean the brittleness reflects how the label is read, not distraction.

Check three: does the abstention signal earn its cost? The use claim fails if rewording disagreement does not beat both maximum probability and the matched-cost ordering ablation under heavy noise.

Check four: does domain pretraining help? Comparing slopes of PubMedBERT and BioLinkBERT against DeBERTa-MNLI and the reranker answers whether domain pretraining reduces overlap-driven distraction. No prior work in the file answers this. If the slopes come out the same, that is also informative: it would mean the problem lives in NLI-style tuning, as Ma et al. suggest, not in the base vocabulary.