Models trained on scientific text get better at scientific tasks. Nobody knows whether they learn the causal claims in that text, such as "drug X lowers blood pressure, which reduces stroke risk," or only learn which technical terms tend to appear together. This proposal answers the question with a controlled experiment rather than a benchmark score. We continue pre-training the same small language model on several token-matched corpora. Each corpus removes exactly one ingredient of scientific writing. One keeps real biomedical abstracts in their natural sentence order. One keeps the same abstracts with sentences shuffled, so word co-occurrence survives but the precondition-to-intervention-to-outcome order is destroyed. A third is "non-research" scholarly documents (journal mastheads, conference front matter, committee rosters), which look like science, with DOIs, affiliations and technical vocabulary, but contain no causal claims at all. We then ask, inside the network, whether each model's hidden states actually carry the information it uses to answer questions about interventions.
The two-sentence version is this. Non-research documents and sentence-shuffled abstracts are natural "placebo corpora" that separate three things a domain model might learn: scholarly surface form, term association, and ordered causal discourse. Measuring each model with causal mediation probes, not just accuracy, tells us which ingredient produces internal causal structure. That in turn tells us whether domain models are better reasoners or only better-informed "causal parrots."
Core Claim
We hypothesize that whatever internal causal structure domain pre-training induces comes mainly from the ordered discourse of empirical abstracts, not from domain vocabulary or scholarly formatting. If so, three things should hold.
First, a model trained on ordered abstracts should show a larger necessary indirect effect, meaning a larger share of its answer that actually passes through the probed hidden state, for intervention outcomes. We measure this on synthetic biomedical-flavored causal stories. The shuffled-abstract model should show roughly the same domain perplexity gain but a smaller indirect effect. The non-research-document model should show neither.
Second, replacing increasing fractions of research text with non-research documents should erode causal mediation faster than it erodes domain perplexity.
Third, ordinary behavioral accuracy on causal benchmarks will be too close to chance at this scale to see any of this. Ordinary linear probes will see it but will confuse ordered-discourse models with association-only models.
The claim is falsifiable in the obvious direction. If the shuffled and ordered models are indistinguishable under mediation analysis, then domain pre-training's causal benefit is association all the way down.
Why This Gap Exists
Several papers motivate the question.
- A Critical Review of Causal Reasoning Benchmarks for Large Language Models catalogues 39 causal benchmarks. It shows that most can be solved by retrieving memorized associations, which is the "causal parrot" failure.
- CLadder generates causal stories from known graphs across Pearl's three rungs (association, intervention, counterfactual). It finds that models collapse on anti-commonsensical stories, where memorized priors point the wrong way.
- CLEAR finds models answering advanced adjustment-set questions while failing the basic path questions those answers depend on.
All three study prompted general-purpose models. None asks what the training corpus contributed.
On the domain side, BioBART continued pre-training BART on 41GB of PubMed abstracts. It reported downstream gains, and incidentally found that the standard sentence-permutation objective hurt biomedical performance. This is the observation we build on. BioBART measured summarization and entity scores, not internal representations, and its corpus was not curated against non-research artifacts.
We Need Improved Data Curation and Attribution in AI for Scientific Discovery documents how much low-integrity material enters scientific corpora. It does not measure what that material does to a model.
The non-research documents are easy to find in any scholarly index. Examples include IEEE Transactions on Fuzzy Systems publication information, conference Front Matter, and committee pages like DSML 2023 Committee. We do not treat them mainly as a contamination problem to be filtered; a title regex probably handles that. We use them as something more useful: a corpus that matches scientific surface form while containing zero causal content.
The measurement method comes from Latent Causal Probing. That work writes a structural causal model of the data-generating process and computes necessary indirect effects to separate what the language model encodes from what the probe learns on its own. It contrasts these against permuted-dynamics baseline graphs. It was only run on a synthetic grid-world programming language with models trained from scratch, and we carry it to domain pre-training of a real base model.
Two papers constrain the design.
- Causal Articulation Theory shows that a model's verbal explanations are often unfaithful rationalizations. We therefore never use generated explanations as evidence.
- The saliency-limitation literature, for example NBDT, shows that attribution maps reveal where a model looks, not what it treats as a cause. Saliency therefore enters only as a baseline we expect to fail.
Research Program
Corpora. Starting from one public base checkpoint (a GPT-2-medium-class decoder, roughly 350M parameters), we run continued pre-training under five arms. All arms see an identical token budget, at a scale set by how many clean non-research tokens we can collect, with an expected order of a few hundred million:
- Ordered abstracts: biomedical abstracts in natural order.
- Shuffled abstracts: the same abstracts with sentence order permuted within each document. This keeps every sentence and therefore every within-sentence claim.
- Non-research documents: format-matched non-research scholarly documents, assembled from mastheads, front matter, committee rosters, calls for papers, and memorials. These are drawn from scholarly indexes and deduplicated.
- General text: general web text.
- No continued training.
A dose-response series then replaces 10%, 30% and 50% of the ordered-abstract tokens with non-research documents. Three seeds per arm.
Probe suite. We extend CLadder's oracle story generator to a biomedical vocabulary. We sample small causal graphs over drug, mediator, biomarker and outcome nodes and render them as abstract-style narratives. Each graph gets a commonsensical rendering, an anti-commonsensical rendering (edges contradict real pharmacology), and a nonsense-term rendering. For each story we pose observational, interventional and counterfactual queries, whose answers the oracle computes exactly.
Mediation measure. Following Latent Causal Probing, we train a probe $g$ on hidden states $h_\ell$ at layer $\ell$ to predict the oracle's interventional answer $Y$. Our score is the necessary indirect effect:
$$\mathrm{NIE}_\ell = \mathbb{E}\big[\,\mathrm{acc}(g \mid h_\ell \text{ from the true graph}) - \mathrm{acc}(g \mid h_\ell \text{ from a permuted-dynamics graph})\,\big],$$
Here the second term retrains the same probe architecture on stories whose surface text is unchanged but whose answers come from a randomly rewired graph. A positive gap means the representation carries the specific causal structure, beyond what probe capacity and lexical cues can supply.
We complement this with activation patching. We swap the hidden state at the mediator token between two stories that differ only in one edge, and record whether the model's next-token answer flips as the oracle predicts. That is a direct interventional test on the network rather than on its output text.
Geometry. We also track how the intrinsic dimensionality of hidden states changes across layers. Joshi et al. report that this follows an expand-then-contract "hunchback" across 28 models. We test whether non-research dilution flattens it, as a cheap correlate of the mediation results.
Baselines
The naive reference point is the base checkpoint with no continued pre-training, plus the token-matched general-text arm. These fix how much causal mediation is already present and how much any extra training adds regardless of domain. Every other arm must exceed the general-text arm's NIE by more than the seed-to-seed spread to count as a domain effect.
The strongest existing behavioral reference is prompted evaluation on CLadder. CLadder reports GPT-4 with its chain-of-thought prompt at 70.40% accuracy, and 6.7B LLaMA and Alpaca at roughly 44–45%, near random. CLEAR reports GPT-4 at 60.5% average and most models under 40% on intermediate and advanced graph tasks. Those numbers say our 350M models will sit at chance behaviorally. We will still run every arm on CLadder, CLEAR and the CausalBench text subset with identical prompts, because behavioral separation would be the cheapest possible answer.
The strongest existing domain reference is BioBART against BART. BioBART reports MeQSum Rouge-L of 53.15 versus 51.05. We run our probe suite on both released checkpoints as an uncontrolled pilot, then check whether our controlled ordered-abstract arm reproduces the direction of that gap on MeQSum, so our corpora are not unrepresentative.
The strongest existing interpretability competitor is a standard linear probe's raw accuracy on the same hidden states, without the permuted-graph control. We also run integrated-gradients attribution of the answer token onto story tokens, scored by how much attribution mass falls on true parent nodes. Both get the same layers, data and hyperparameter search budget as the NIE probe.
The NIE "beats" these baselines only if two conditions hold. It must separate the ordered and shuffled arms when they do not, with a non-overlapping 95% bootstrap interval across seeds. Its per-checkpoint value must also predict held-out activation-patching flip rates better, measured by higher Spearman correlation across all arm and seed checkpoints.
The key ablations are the shuffled arm, which isolates discourse order, and the non-research arm, which isolates scholarly form. The 0–50% dilution series tests whether any effect is graded.
No reported result was found on Lacuna for necessary indirect effects on any real pre-trained language model. The NIE comparison is therefore new measurement, not a race against a published number.
Why not just…?
Why not just read off behavioral accuracy on CLadder for each arm? At this model size the reported numbers put models near chance, around 44–45% for 6.7B models in CLadder. A behavioral comparison of 350M models will almost certainly be a floor effect. This is a real threat only if behavior does separate the arms. We run it first, and if it separates them as cleanly as the mediation measure, we report that the cheaper measure suffices and narrow our claim to the cases where the two disagree.
Why not just use an ordinary probe or a saliency map? An ordinary probe can learn the task from lexical cues itself, which is exactly the confound Latent Causal Probing was built to remove. A saliency map shows which tokens mattered, not whether the model represents the graph that makes them matter, as NBDT argues for vision. The shuffled-abstract arm is the test case. It preserves term co-occurrence, so we predict ordinary probes and attribution will score it similarly to the ordered arm, while NIE and patching will not. If they score the two arms the same too, that prediction fails publicly.
Why not just filter non-research documents with perplexity or a title rule and measure downstream scores? A title or metadata rule likely does filter most of them. That is why this proposal is not a filtering paper. Perplexity from PLM Is Unreliable shows repetition dropping GPT2-large perplexity from 411.99 to 4.84, so perplexity would favor list-like mastheads. Either way, filter quality is beside the point here, because we use these documents deliberately as a placebo. The dilution series does give a practical byproduct: how much contamination it takes before causal mediation degrades.
Why not just compare off-the-shelf BioBART and BART? Those models differ in data, size, objectives and training length, so any difference is confounded. We run that comparison as a pilot, but only token-matched arms from one checkpoint can attribute an effect to a corpus property.
Decisive Checks
The study succeeds as science whichever way it comes out, provided four checks are clean.
- The shuffled and ordered arms must reach comparable domain perplexity on held-out abstracts. Otherwise any NIE gap is just more domain learning.
- On the ordered arm, the permuted-graph control must drive NIE to near zero on nonsense-term stories. Otherwise the probe is exploiting surface artifacts of the generator.
- NIE on anti-commonsensical stories is the critical cell, because there memorized pharmacology actively misleads. A positive ordered-minus-shuffled gap there is the strongest evidence that ordered discourse teaches structure rather than recalled facts.
- Activation patching must agree in direction with NIE across checkpoints. Disagreement means the probe measures something the model does not use.
If ordered abstracts win on all four, the practical message is that scientific pre-training corpora should preserve discourse order and exclude format-only documents. If shuffled abstracts match them, domain models are better-read parrots, and causal reasoning must come from somewhere other than the corpus.
References
- A Critical Review of Causal Reasoning Benchmarks for Large Language Models
- CLadder: A Benchmark to Assess Causal Reasoning Capabilities of Language Models
- CLEAR: Can Language Models Really Understand Causal Graphs?
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- We Need Improved Data Curation and Attribution in AI for Scientific Discovery
- IEEE Transactions on Fuzzy Systems publication information
- Front Matter
- DSML 2023 Committee
- Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data
- Causal Articulation Theory (CAT): Articulating Static and Temporal Causal Models Beyond LLM Rationalizations
- NBDT: Neural-Backed Decision Tree
- Geometry of Decision Making in Language Models
- Perplexity from PLM Is Unreliable for Evaluating Text Quality