Counterfactual Results Twins for Biomedical Pre-Training

Biomedical language models learn their field's vocabulary well, but they also learn its habits. Most published abstracts report positive findings, so a model trained on PubMed can learn that a study about a drug probably concludes the drug works, without reading the results. This proposal builds pre-training data that breaks that habit. For a sample of abstracts, we write a "counterfactual twin": the same background, population and methods, but with the reported result changed (an effect reversed, a p-value moved across the significance line, a confidence interval made to cross the null), and the conclusion rewritten to match. Pre-training on factual–counterfactual pairs makes the conclusion depend on the results text, because within a pair the results are the only thing that differs.

The question is not just whether this helps. It is whether doing it during pre-training buys anything that cheaper fixes do not. Those fixes include counterfactual augmentation applied only at fine-tuning time, re-weighting the corpus toward negative studies, or extra masking on conclusion sentences. We test this on PubMedQA, where 96.5% of questions require quantitative reasoning and BioBERT trails human experts by ten points. We also test on a new contrast set of minimally edited PubMedQA contexts, where a shortcut-following model should fail.

Core Claim

Domain pre-training passes on the confounding in the corpus. In biomedical abstracts, publication bias links a study's topic and framing to the polarity of its conclusion. A small fraction of results-intervened twins, mixed into continued in-domain pre-training, should reduce how much the encoder relies on that link. This should show up most clearly on inputs that differ only in their results. The gain should exceed both fine-tuning-time counterfactual augmentation and pure re-weighting at matched compute. If it does not, the pre-training stage is the wrong place for counterfactuals, and that negative result is itself useful.

Prior work motivates this claim but does not test it. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing shows that pre-training BERT from scratch on PubMed with a domain vocabulary beats general-domain initialization (PubMedBERT, 81.16 on BLURB). Its objective, however, is ordinary masked-word prediction over observational text.

Two papers from clinical NLP show what that leaves behind. Where do Clinical Language Models Break Down? finds that ClinicalBERT masters jargon but stumbles on numbers and negation. MedNLI Is Not Immune shows that a hypothesis-only classifier reaches 62.6% on MedNLI, and that full models fall to about 40% on the filtered hard subset. The general argument of Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? points the same way: next-token training captures word co-occurrence rather than event causality. Results-to-conclusion is exactly such a causal dependency.

Relation to Closest Work

Three papers come closest, and each differs from this proposal on one axis.

Pulling Up by the Causal Bootstraps debiases data before training by re-weighting and re-sampling under an interventional distribution, on image and tabular tasks. It never creates new inputs. We create new text, and we keep a re-weighting arm so we can tell whether new textual support matters.

Causal Document-Grounded Dialogue Pre-training adds counterfactual-style objectives to pre-training. One trains invariance to inserted noise; the other trains the model to avoid a response once its evidence is deleted. Its interventions are coarse (insert or delete sentences) and its data is open-domain Wikipedia and Reddit. We make targeted, meaning-changing edits to one variable, the reported result, inside a specialized corpus, and we require the conclusion to change with it.

Dually Self-Improved Counterfactual Data Augmentation (DICT) and Iterative Counterfactual Data Augmentation generate label-flipping counterfactual edits, but only for downstream classifiers on general NLI or review data. We reuse their idea of verified minimal edits as a method, and we treat fine-tuning-time augmentation as the main competitor.

CausaLM intervenes in embedding space rather than on text. We do not adopt it, because a latent intervention cannot be fed to a standard token-level pre-training pipeline.

Research Program

The causal story is deliberately small. An abstract has four parts: its background and design ($B$), its reported results ($R$), its conclusion ($C$), and a publication-process confounder ($U$) that links $B$ to the polarity of $C$. Only $R$ should cause $C$; $U$ and $B$ should not.

Twin generation follows the abduction–intervention–prediction recipe in its simplest textual form. Abduction means keeping $B$ verbatim. Intervention means a rule-based edit to $R$: flipping direction words ("increased"/"decreased", "higher"/"lower"), changing a p-value from 0.03 to 0.41, moving a confidence-interval bound across 1.0, or negating "significant." Prediction means rewriting $C$ to the polarity the new $R$ implies, using a constrained instruction-tuned rewriter.

Because rule edits to statistics can be checked mechanically, each twin passes two filters. First, a deterministic check confirms that the edited statistics are internally consistent. Second, an independent conclusion-polarity classifier confirms that the new conclusion flips. Abstracts whose results cannot be parsed into editable statistics are skipped.

As in Selecting Data Augmentation for Simulating Interventions, we expect indiscriminate edits to hurt. We therefore restrict interventions to the results variable, and we audit a sample of 500 twins by hand with biomedical annotators. We report the audit's error rate as a measured fact rather than assuming it.

Training is continued in-domain pre-training from the PubMedBERT checkpoint. Every arm gets the same number of additional tokens and steps, so the arms differ only in what they read. In the twin arm, a fraction $\rho$ (swept over 2%, 5% and 15%) of sequences are factual–twin pairs. Conclusion spans are masked in both members of a pair. The loss is ordinary masked-word prediction plus a pair term:

$$\mathcal{L} = \mathcal{L}_{\text{MLM}} + \lambda \, \max\big(0,\ m - \log p_\theta(C \mid B, R) + \log p_\theta(C \mid B, R')\big)$$

Here $R'$ is the edited result, $C$ is the factual conclusion, $p_\theta$ is the model's pseudo-likelihood of the masked conclusion tokens, $m$ is a margin, and $\lambda$ weights the term. Read in words, the pair term asks that the factual conclusion be more probable under the factual results than under the counterfactual results. The symmetric term for the twin conclusion is included as well.

Setting $\lambda=0$ gives the "twins as plain data" variant. Comparing it with the full objective shows whether the explicit contrast matters or the data alone is enough.

Baselines

The do-nothing reference is the PubMedBERT checkpoint fine-tuned directly. Beside it sits a compute-matched control: continued pre-training on an equal number of extra real PubMed abstracts. That control separates "more training" from "counterfactual training." For anchoring, PubMedQA reports 68.1% accuracy for BioBERT with multi-phase fine-tuning, a 55.2% majority baseline, and 78.0% for human experts. The only PubMedBERT figure available is the aggregate 81.16 BLURB score; no reported result found on Lacuna for PubMedBERT on PubMedQA alone. We will reproduce that number ourselves under our fine-tuning protocol.

The strongest existing competitor is fine-tuning-time counterfactual augmentation in the style of DICT. It uses the same twin generator, applied to the PubMedQA and MedNLI training contexts, with DICT's balanced loss, on top of the continued-pre-training control. DICT reports an F1 of 64.89% versus 58.29% for base RoBERTa on SNLI-1. It is the strongest competitor because it targets the same shortcut at a fraction of the cost.

To make the comparison fair, both arms receive the same total number of counterfactual examples generated by the same pipeline, and the same hyperparameter search budget of 20 trials over learning rate, epochs and loss weight. Our method beats it if it is better on the contrast set by more than the seed-to-seed spread (five seeds, paired bootstrap), while staying level on standard PubMedQA and BLURB.

The third baseline is causal-bootstrapping-style re-weighting in the spirit of Pulling Up by the Causal Bootstraps. Abstracts are resampled during continued pre-training so that conclusion polarity is balanced within each topic cluster; we use MeSH headings as a stand-in for the confounder. No new text is created.

The key ablation is conclusion-targeted masking without twins. It applies the same masking of conclusion spans and number tokens to real abstracts only. This isolates whether the counterfactual pairing, rather than the masking pattern, does the work. The $\lambda=0$ variant is a second ablation on top of it.

Why not just…?

Why not just do counterfactual augmentation at fine-tuning time? This is the real threat, and we do not dismiss it. DICT and ICDA show that task-level counterfactuals reduce shortcut use. If the downstream training set already contains enough twins, pre-training may add nothing. We expect pre-training to matter in two cases: when labeled data is scarce (PQA-L has only 1,000 expert labels), and on tasks the twins never touched, such as MedNLI and the BLURB tasks. We test this directly. The fine-tuning-augmentation arm runs at 100, 300 and 1,000 labeled examples. A transfer comparison asks whether gains from twins built only on PubMed survive in clinical inference. If fine-tuning augmentation matches us at every size and on transfer, we will say that pre-training-stage counterfactuals are not worth their cost.

Why not just re-weight the corpus toward negative studies? Re-weighting corrects the marginal link between topic and conclusion polarity. It does this cheaply, and in Pulling Up by the Causal Bootstraps it is robust on images. It never shows the model two texts identical except for their results, however, so it gives no direct signal about which tokens carry the cause. It is also limited by how many genuinely negative abstracts exist on each topic. The re-weighting arm settles this question. If it matches twins on the contrast set, textual support is unnecessary and the cheaper method wins.

Why not just mask conclusions and numbers more aggressively? Targeted masking could, by itself, push the encoder to attend to results; whole-word masking already matters in domain pre-training. But in observational text the masked conclusion remains predictable from topic priors, so masking alone need not cut the shortcut. The masking-only ablation measures this directly.

Why not just use a larger general model with prompting? CausalProbe-H shows that frontier models falter on fresh, deceptive cause–effect pairs, with Claude 3 Opus below 70%. On grounded dialogue, prompted GPT-3.5 reaches 3.57 BLEU against 43.0 for causally pre-trained CausalDD. We include one prompted instruction model on the contrast set as a reference point, not as the main comparison.

Proposed Experiments and Decisive Checks

The central new evaluation is a PubMedQA contrast set. For 300 PQA-L test contexts, biomedical annotators write minimal edits to the results that flip the yes/no answer, and we keep the originals alongside. The headline metric is pair consistency: both members of a pair must be answered correctly. Accuracy alone is not enough, because a model that always says "yes" gets one member of each pair right.

Annotators write these edits by hand and independently of the training generator. This guards against rewarding a model that has merely learned the generator's edit style. As a further check, we report results separately for edit types the generator never produces, such as changes to group sizes or follow-up periods.

We also report PubMedQA PQA-L accuracy and the full BLURB suite, to show that domain skill is not lost. For transfer, we report MedNLI accuracy, where ClinicalBERT scores 80.17%, together with accuracy on the AFLite hard subset described in MedNLI Is Not Immune.

A shortcut diagnostic tracks reliance on the question alone. We fine-tune each encoder with the abstract hidden, giving it only the question (the title). A model that has stopped using topic priors should gain less from the question alone, and its accuracy with the full context should rise.

The proposal succeeds if three things hold at once. Twin pre-training must beat all three non-trivial baselines on pair consistency in the low-label regime. It must not lose more than half a point on BLURB. And its advantage over fine-tuning augmentation must survive transfer to MedNLI's hard subset.

Two outcomes would refute it. If re-weighting or masking-only matches twins, textual counterfactuals are unnecessary. If fine-tuning-time augmentation matches twins at every labeled-data size, the benefit belongs to the task stage, not the pre-training stage. Finally, a dose-response sweep over the twin fraction $\rho$ checks whether gains grow and then flatten, or whether heavy twin mixing begins to distort the corpus. This is the selection-bias risk that counterfactual upsampling carries.