Shortcut Detectors That Know Medicine: Domain-Aware Bias Experts for Clinical Inference

Clinical language models score well on MedNLI, a benchmark where a model reads a sentence from a patient's chart and decides whether a physician-written statement follows from it, contradicts it, or neither. Much of that score may be cheating. A model that never sees the chart and reads only the physician's statement already reaches 62.6% accuracy against a 33.3% chance level. It succeeds because physicians who wrote the statements followed predictable habits. They swapped a drug for its drug class to make entailments, wrote "the patient has normal vitals" to make contradictions, and guessed common risk factors such as smoking for neutral cases (MedNLI Is Not Immune). The usual fix in general-domain language understanding is residual debiasing. A deliberately limited "bias expert" is trained to learn the cheap shortcuts, and the main model is trained only on what that expert cannot explain. The automated versions of this fix assume that shortcuts are whatever a small, weak model picks up first. In medicine that assumption breaks: knowing that clindamycin is an antibiotic is simple, but only for a model that has read clinical text.

The proposal is to build the bias expert from the domain-pretrained model's knowledge of words while stripping away its ability to combine them. The expert is a bag-of-tokens classifier over frozen clinical embeddings, with no attention and no composition across words. It can recognize clinical lexical relations, but it cannot do inference. The claim is that for expert-annotated domain data, a good shortcut detector is defined by limited composition, not limited size or vocabulary. On this view, domain pre-training is the ingredient that makes residual debiasing work in specialized domains, rather than a rival to it. We test this on MedNLI with Clinical BERT, against generic weak learners, hypothesis-only bias models, and data filtering. We also probe whether the debiased model has learned clinical inference or has only learned to subtract the shortcuts.

Core Claim

Residual debiasing combines a frozen bias expert and the main model by adding their logits during training. In the Product of Experts form (Don't Take the Easy Way Out) and the residual-fitting form DRiFt (Unlearn Dataset Bias in NLI by Fitting the Residual), the main model's gradient shrinks on examples the bias expert already gets right confidently. The main model therefore learns mostly from counterexamples, and the bias expert is discarded at test time. Everything depends on what the bias expert can and cannot represent.

Two automated designs exist, and both define "biased" as "learned by a weak model." Learning from others' mistakes uses a 2-layer, 4M-parameter TinyBERT as the weak learner. Towards Debiasing NLU Models from Unknown Biases uses a shallow model trained briefly on about 2,000 examples. Both were validated only on general text, where shortcuts are lexical overlap and negation words. MedNLI's shortcuts are different. Its classes have matched sentence lengths, and its heuristics are semantic: drug-to-class hypernyms, "everything is fine," and plausible comorbidities. A general-vocabulary TinyBERT should mostly miss them. A truncated or early-stopped Clinical BERT may catch them, but it may also learn real inference, and the main model is then penalized for learning real inference.

We define the bias expert by what it is forbidden to compute. Let $e(t)$ be the frozen input-layer embedding that Clinical BERT assigns to token $t$, and let $p$ and $h$ be the premise and hypothesis. The expert is

$$f_b(p,h) = \mathrm{MLP}\big(\,\mathrm{mean}_{t\in h}\, e(t),\ \max_{t\in h,\,s\in p} \cos(e(t),e(s)),\ \mathrm{hist}_{t\in h}\big[\max_{s\in p}\cos(e(t),e(s))\big]\big).$$

The first term carries hypothesis-only cues such as "normal" or "smoking." The second and third terms carry word-level premise–hypothesis relatedness, such as the clindamycin/antibiotics link. Here $\mathrm{hist}$ is a fixed-bin histogram of best-match similarities. The expert has no word order and no contextual composition, so it cannot track negation scope, temporality, or which patient finding a statement refers to. The main model, Clinical BERT, is trained with the Product of Experts loss plus the auxiliary cross-entropy term of Sanh et al. During training the prediction is $\mathrm{softmax}(f_b + f_d)$ with $f_b$ frozen, and only $f_d$ is used at test time.

There is an obvious objection: the hypernym habit is sometimes valid medicine. Clindamycin really is an antibiotic. The method does not stop the main model from learning that fact. It stops the main model from relying on that fact alone when the bias expert already predicts entailment confidently. Examples where the class-level statement is contradicted by context, such as "the patient was taken off antibiotics," keep their full gradient. Whether this distinction holds in practice is the central empirical question, and the probing below tests it directly.

Research Program

The work has three parts.

The first part is diagnostic and has not been reported. Herlihy and Rudinger built an adversarially filtered "difficult" MedNLI split with AFLite, a procedure that repeatedly removes examples that simple classifiers can predict. They evaluated fastText and generic models on it. Clinical BERT reports 85.4% on standard MedNLI (Publicly Available Clinical BERT Embeddings), and the two-stage BioBERT pipeline of KU_ai at MEDIQA 2019 reports 85.8%, but neither was evaluated on hard or hypothesis-only partitions. We re-run AFLite following the published procedure, since the file does not indicate a released split. We then measure general BERT, BioBERT, and Clinical BERT on standard, easy, and difficult subsets. This answers whether domain pre-training buys reasoning or more efficient shortcut use. The approximately 51% and 40% full-model figures in that paper come from weaker models and are not comparable to the 85% numbers, so all reference points will be re-measured.

The second part is the method comparison. Bias experts are ordered by what they can see. A general-vocabulary TinyBERT weak learner is the generic weak learner. A truncated two-layer Clinical BERT weak learner gives domain knowledge together with composition. A hypothesis-only clinical model gives domain knowledge without the premise. The proposed bag-of-tokens expert gives domain knowledge and word-level cross-sentence features without composition. For each expert, we first record how well it alone predicts each heuristic category, labeled using the pointwise mutual information and MeSH-based categorization of Herlihy and Rudinger. We then train Clinical BERT against it.

The third part checks representations. Debiasing Methods Make Bias More Accessible used minimum-description-length probing, which measures how compactly a feature can be decoded from a model's internal states. They showed that debiased general models encode shortcuts more extractably, with a 0.752 correlation between HANS gains and overlap extractability. This finding constrains the proposal. If our gains come only from the output layer learning to subtract the shortcut, we expect the same pattern. We therefore probe each debiased encoder for each clinical heuristic: hypernym match, normal-finding language, and risk-factor terms. We also probe for legitimate clinical knowledge, specifically drug-class and finding-type labels derived from MeSH. The hope is that clinical knowledge stays as extractable as in the ERM model, where ERM means empirical risk minimization, or ordinary training on the labeled data. A drop would mean debiasing damaged medical knowledge. A rise in heuristic extractability would replicate the paradox in the clinical setting, which is itself a reportable result.

Baselines

The naive reference point is Clinical BERT fine-tuned with ordinary cross-entropy on MedNLI, as in Publicly Available Clinical BERT Embeddings, which reports 85.4% accuracy on the standard test set. This is the strongest reported in-distribution number in the file for a single domain model. The KU_ai two-stage pipeline (MultiNLI and SNLI, then MedNLI, with BioBERT) reports 85.8% test accuracy for the single model and 86.1% development accuracy for its ensemble (KU_ai at MEDIQA 2019). It also strips de-identification tags. We reproduce it, with the same tag stripping applied to every system, because it is the strongest pipeline that does not debias. No reported result was found on Lacuna for any residual or Product of Experts debiasing method on MedNLI or its difficult split, so all debiasing baselines are run by us.

The strongest existing debiasing competitors are three. The first is Sanh et al.'s weak-learner Product of Experts with the auxiliary cross-entropy term, run with a general TinyBERT; on HANS it gained 24 points on non-entailment cases. The second is Utama et al.'s shallow-model self-debiasing, which gained 8.2 points on HANS. The third is saliency-gated Product of Experts (Guide the Learner). That method relaxes the penalty when the bias expert and main model attend to different tokens, and it reported 68.06% versus 62.85% on HANS. It is the most direct competitor on the "shortcut overlaps with real knowledge" problem. We also include DRiFt and Product of Experts with a hypothesis-only Clinical BERT bias expert, the known-bias design adapted to the domain, and a model trained on AFLite-filtered data.

All systems share the Clinical BERT main model, the same tag-stripped data, and the same seeds; we use at least five seeds and report means with intervals. Each method gets an equal hyperparameter budget. That budget covers learning rate and epochs, plus for weak learners their depth and training size, the sensitivity flagged as a known weakness. Model selection uses the standard development set only, because selecting on a hard split would leak.

The key ablations isolate the proposal's ingredient. The first swaps the clinical embeddings in the bag-of-tokens expert for general BERT embeddings, which removes domain knowledge and keeps the architecture. The second removes the cross-sentence similarity features, which reduces the expert to hypothesis-only. Beating the baselines means two things together. Accuracy on the AFLite-difficult split and on a counterexample set must improve significantly over the best debiasing baseline. Standard-test accuracy must stay within about one point of the 85.4% ERM reference, since a large in-distribution drop is the known failure mode of these methods.

Why not just…?

Why not just filter the shortcut examples out with AFLite and train on the rest? MedNLI is small. Herlihy and Rudinger describe AFLite as a reactive diagnostic, and He et al. argue that filtering discards useful signal that residual fitting keeps through gradient reweighting. That is an argument, not a measurement, so we include AFLite-filtered training as a baseline. If it matches our method on the difficult split at similar standard accuracy, the simpler method wins and we say so.

Why not just use a hypothesis-only clinical bias model? This is the real threat, because the "everything is fine" and risk-factor heuristics are hypothesis-only. The hypernym heuristic, however, is a relation between a premise word and a hypothesis word, and a hypothesis-only model cannot see it. The prediction is therefore specific. Our expert should beat the hypothesis-only expert mainly on hypernym-category hard examples and roughly tie elsewhere. If it ties everywhere, the cross-sentence features add nothing, and the contribution reduces to the diagnostic and probing findings.

Why not just use Sanh et al.'s generic weak learner, or a truncated Clinical BERT as the weak learner? The general TinyBERT likely lacks the vocabulary to find clinical shortcuts. The truncated Clinical BERT has the vocabulary but also attention across words, so it may learn genuine inference and suppress the wrong examples. Both are run. Measuring each expert's standalone accuracy per heuristic category, and its accuracy on examples that require composition, such as negated or temporal statements, shows directly which experts absorb shortcuts and which absorb reasoning.

Why not just rely on better clinical pre-training or larger models? The domain literature shows targeted pre-training helps; the file cites MonoCoder and BERTHop as examples. Nothing in the file shows that pre-training removes annotation shortcuts. BiasTrojan shows that large models can be skewed by a few hundred biased examples. The first-stage diagnostic comparing general BERT, BioBERT, and Clinical BERT on the difficult split answers this question empirically.

Proposed Experiments

The datasets are MedNLI's standard split, our reproduced AFLite easy and difficult splits, and per-heuristic subsets labeled with the Herlihy and Rudinger categorization. We also build a small physician-checked counterexample set of a few hundred pairs, edited from MIMIC-III premises, in which each heuristic points the wrong way. Examples include a drug-class statement contradicted by context, an "is stable" statement that is actually entailed, and a risk-factor statement that the chart entails. The metrics are accuracy per split and per heuristic, standalone accuracy of each bias expert, and minimum-description-length compression for heuristic and knowledge probes.

As a sanity check that the design is not tuned only to MedNLI, we run the same bag-of-tokens expert with general BERT embeddings on MNLI and test on HANS. It should roughly match the generic weak learner there, because general shortcuts do not need specialized vocabulary.

Three outcomes would decide the question. First, the clinical-embedding expert beats its general-embedding ablation on the difficult and counterexample sets. That would show domain pre-training is what makes the bias expert work. Second, it beats the hypothesis-only expert specifically on hypernym cases. Third, the probes show clinical-knowledge extractability preserved at the ERM level, which separates real gains from output-layer subtraction. If the first fails, the central claim is wrong. If the third fails, we report a clinical instance of the robustness paradox rather than a robustness method.