Perturbation Screens as Distant Supervision for Reading Causal Claims in Biology Papers

Genomics now produces far more candidate genes than anyone can test. A study of stress-responsive gene expression or a genetic mapping study can flag hundreds of genes, and most of them are bystanders, meaning genes that respond to a trait without driving it. The literature is full of sentences about such genes. Some report what happened when a gene was knocked out or overexpressed ("knockdown of X abolished Y induction"). Others only report that two genes moved together ("X and Y were co-upregulated under cold"). We propose to teach a text model to tell these apart. Our labels do not come from human annotators or from a curated knowledge base. They come from the outcomes of large pooled perturbation experiments, in which thousands of genes are each perturbed and the effect on every other gene is measured in single cells.

The idea is a change of label source for distant supervision. Distant supervision is the standard trick of labeling text automatically by matching it against an existing database: if the database says A relates to B, every sentence mentioning both is treated as a positive example. In our version, the "database" is a perturbation screen. A sentence co-mentioning a transcription factor and a target gene is labeled positive if perturbing that factor actually changed that gene in the screen, with the sign of the change recorded. It is labeled negative if the factor was perturbed effectively and the gene was expressed but did not move. We pair these labels with a sentence encoder that routes information along typed, directed grammatical links (subject, object, negation, modifier), not along an undifferentiated graph. The reason is that the cues separating interventional claims from correlational ones are structural. They depend on which gene is the grammatical subject of the perturbation verb, whether a negation attaches to it, and the direction of the regulation. The final test is downstream. The extracted literature graph replaces a computationally inferred regulatory network as the prior in a causal model of single-cell perturbations, and it is evaluated on a different, held-out screen.

Core Claim

Text labeled by experimental outcomes teaches a different and more useful skill than text labeled by curated relations. A model trained this way should rank literature-supported regulator–target pairs by whether they hold up under intervention, not by how often they are asserted. We also hypothesize that typed, directed dependency structure carries much of this signal once the memorization shortcut is removed. That second claim is a separable, falsifiable hypothesis and not an assumption. The first claim stands or falls on whether the resulting prior beats both a computational regulatory-network prior and a prior mined by a strong supervised extractor of typed, signed relations when predicting unseen perturbation outcomes.

Why this is not already done

The closest work sits on two separate edges of the problem and does not join them. MedDistant19 (Amin et al.; Amin et al.) aligns 32 million MEDLINE abstracts to the SNOMED CT clinical ontology. Its key finding is that earlier distantly supervised biomedical benchmarks leaked 26–86% of their test triples into training, and that a top model's AUC fell from 82.6 to 11.8 once leakage was removed. This motivates our insistence on splits with no shared entities. Its labels, however, are clinical ontology facts, its encoders are flat sequence models, and its failure analysis points to inverse and directional relations as a weakness.

RegulaTome (RegulaTome) supplies manually annotated typed, directed, signed regulatory relations and mines 18 million of them from PubMed with a RoBERTa model. It is our strongest competitor for producing a literature prior. Its labels record what annotators read in the text, though, not what experiments later confirmed, and it never checks its relations against perturbation data.

On the biology side, gCRL (Lagani et al.) shows that regulatory-network priors built with CellOracle improve prediction of unseen double perturbations. The review Causal machine learning for single-cell genomics explicitly notes that no automated method exists to populate such priors from the experimental literature. Neither uses text.

Our proposal reverses the usual flow. It uses a perturbation screen to label text, then uses the trained text model to build a prior for a different screen.

The syntax component reuses established machinery. Type-aware graph convolution (Tian et al.) and Universal-Syntax GCN (Fei et al.) put arc labels and directions into message passing and showed robustness gains in sentiment analysis. They have never been tested in biomedical distant supervision. Pruned-tree graph convolution (Zhang et al.) showed that keeping tokens within one hop of the shortest dependency path preserves negation, and that dropping the tree cost 3.2 F1 on TACRED. It does, however, treat all arcs as identical. The Extended Dependency Graph (Peng et al.) constrains our design. Hand-added semantic links helped protein-interaction extraction in some corpora and hurt it in others, so we learn edge-type weights rather than adding heuristic links.

Research Program

Building the labels. The supervision source is the Joung et al. (2023) Perturb-seq screen used in Lagani et al., which profiled 3,266 transcription-factor overexpressions in human embryonic stem cells. For each perturbed factor, we call differentially expressed genes against control cells, keeping the direction of change. A pair is positive if it is significant, and negative if the factor's own overexpression is confirmed and the target is expressed but shows no change. All other pairs are left unlabeled rather than called negative. This guards against the fact, emphasized in Causal machine learning for single-cell genomics, that real perturbations are partial and noisy rather than clean interventions.

We then align these labels to PubMed sentences that co-mention the pair, using gene name normalization, in the style of Mintz et al.. Two kinds of noise are expected:

  • Usual distant-supervision noise. A sentence may mention both genes without stating any relation between them.
  • Cell-context mismatch. A claim made in liver cells may genuinely fail in stem cells.

We measure the second directly. A sample of 500 aligned sentences is hand-coded both for whether the sentence asserts regulation and for the cell context it describes. From this we estimate the label-noise rate per stratum. Following MedDistant19's finding that sentence-level training beat bag-level multi-instance learning, we train at the sentence level and aggregate to pairs by max-pooling.

The encoder. Sentences are parsed with a biomedical dependency parser. A PubMedBERT encoder feeds a type-aware graph convolution over the pruned dependency tree. Each update sums over neighbors $j$ of word $i$:

$$h_i^{(l+1)} = \sigma\Big(\sum_{j \in N(i)} \alpha_{ij}\, W_{t(i,j),\, d(i,j)}\, h_j^{(l)}\Big)$$

The terms are defined as follows:

  • $h_i^{(l)}$ is word $i$'s representation at layer $l$.
  • $t(i,j)$ is the dependency label of the arc, for example a nominal subject or a negation.
  • $d(i,j)$ indicates whether $j$ is $i$'s head or its dependent.
  • $W$ is a weight matrix learned separately for each label–direction pair.
  • $\alpha_{ij}$ is an attention weight computed from the arc label and both endpoint representations, as in Tian et al..

The output head predicts three classes for the ordered pair (factor, target): up-regulation, down-regulation, and no effect. Directionality is therefore part of the task, so the model cannot score well by recognizing that two genes are related.

Splits that defeat memorization. Training, development, and test sets are split by regulator. No test-set transcription factor appears in training sentences as the labeled regulator. Following MedDistant19, we also report an "only mention" probe in which the context is masked and only gene names remain. Any accuracy on this probe measures pretrained memorization rather than reading.

The downstream prior. The trained model scores every factor–target pair mentioned in PubMed abstracts published before 2019. The cutoff excludes text describing the Norman et al. (2019) screen itself. The resulting directed, signed graph replaces the CellOracle network as the prior in gCRL-VAE. That model is then evaluated on unseen double perturbations in the Norman K562 CRISPR-activation screen, which uses a different cell type, perturbation technology, and laboratory from the screen that produced the labels.

Baselines

Naive references. The do-nothing reference for the downstream task is Discrepancy-VAE without any regulatory prior. Lagani et al. report an average Euclidean centroid distance of 4.324 for it on unseen Norman double perturbations, against 5.504 for the worst baseline. For the text task, the naive reference is literature co-occurrence frequency: rank pairs by how many abstracts mention both genes. BioRelEx 1.0 found co-occurrence baselines performed poorly at relation extraction, but it measured agreement with annotators, not with experiments. Co-occurrence could plausibly track well-studied true regulators, so we run it rather than assume it fails.

Strongest existing prior. This is gCRL-VAE with its CellOracle network, which Lagani et al. report at 3.639 average Euclidean centroid distance. We rerun it with the same code, latent size, training epochs, and tuning grid as every literature-prior variant. Beating it means a lower distance on the same held-out double perturbations, with confidence intervals over five seeds that do not overlap.

Strongest literature extractor. The RegulaTome RoBERTa extractor reports 66.6% F1 on its own corpus (RegulaTome). We apply it to the same pre-2019 abstracts to build a signed, directed graph. For fairness, we also fine-tune it on our perturbation labels with the same compute and tuning budget as our model. For the general-purpose flat encoder, MedDistant19 reports PubMedBERT at AUC 0.80 and F1 0.76 at the sentence level (Amin et al.), and Macro-F1 65.4 (Amin et al.). Those numbers are on clinical relations. No reported result exists on Lacuna for any text model scored against perturbation-derived labels, so all text-side comparisons are run fresh.

Ablations. Two ablations isolate the key ingredients.

  • Label-source ablation. Train the identical typed-graph encoder on RegulaTome-style curated labels instead of perturbation labels. This tests whether the experimental-outcome labels, not the architecture, drive downstream gains.
  • Syntax ablation. Compare the typed graph, an untyped graph over the same pruned tree (the Zhang et al. design), and no graph (PubMedBERT alone). All three share one backbone and one tuning budget.

Why not just…?

Why not just fine-tune PubMedBERT on the perturbation labels and skip the parser? This is the most serious threat to the syntax half of the proposal, and it may succeed. The evidence that explicit syntax helps on top of pretrained encoders comes from other domains. In sentiment analysis, Fei et al. report roughly 10% accuracy gains on an adversarial robustness set. In general relation extraction, Zhang et al. report a 3.2 F1 drop without the tree. Neither covers biomedical text, where parse errors are worse. Our reason to expect a gain is specific. The regulator-disjoint split removes the memorization shortcut that Amin et al. exposed, and MedDistant19's failure analysis singles out direction and inverse relations, which arc direction encodes explicitly. The syntax ablation settles the question. If PubMedBERT matches the typed graph under the disjoint split, we report syntax as unnecessary, and the proposal reduces to its label-source claim, which is tested independently.

Why not just use RegulaTome's 18 million mined relations as the prior? This is the natural cheaper path, and it is a real threat to the label-source claim. RegulaTome already captures direction and sign. Its annotators, however, label what a sentence asserts. A correlational claim worded as regulation is a positive example to them, which is exactly the bystander confusion that studies like Wang et al. and Hao et al. leave unresolved. The label-source ablation compares RegulaTome-derived and perturbation-derived priors inside gCRL on Norman. If RegulaTome is as good, the extra machinery is not justified, and we would say so.

Why not just weight pairs by citation or co-mention counts? Counts reward well-studied genes, not causal ones. Genes co-cited with a phenotype are often stress-response bystanders. BioRelEx 1.0 found co-occurrence weak even for annotated binding relations. We still include counts as a downstream prior, because popularity could correlate with being a true hub. Our model must beat counts specifically on less-cited factors (the bottom half by citation count) to show it is reading rather than counting.

Why not just use the stem-cell screen directly as the prior for K562? If the source screen alone transfers as well as the literature graph, text adds nothing. We include the raw Joung-derived network as a prior. The literature graph should add value through coverage: it covers relations and cell contexts beyond the one screen. This shows up as gains on Norman genes whose regulators were weakly perturbed or absent in Joung.

Proposed Experiments

Experiment 1: text-side evaluation. We report three-way macro-F1 and AUC under the regulator-disjoint split for all encoder variants. Alongside, we report the "only mention" probe and per-arc-label attention to check whether negation and subject arcs drive the predictions. As a sanity check that the typed encoder is not weaker on standard data, we train it on MedDistant19's inductive split against the reported PubMedBERT AUC of 0.80 (Amin et al.). We also report results stratified by parse confidence, to measure how much parser error limits the syntax path in biomedical text.

Experiment 2: the decisive downstream test. We run gCRL-VAE on Norman unseen double perturbations with six priors:

  • no prior
  • CellOracle
  • co-occurrence counts
  • the raw Joung network
  • the RegulaTome-mined graph
  • our perturbation-supervised graph

The proposal succeeds only if our graph beats CellOracle's reported 3.639 and also beats the RegulaTome graph under identical training. Two controls test whether the gain comes from real signal. First, a degree-preserving shuffle of our graph's edges should erase the gain. Second, flipping all predicted signs should hurt.

Experiment 3: label leakage. Restricting abstracts to those published before 2019 must not change conclusions relative to a version restricted further to papers that never cite either screen's authors' prior perturbation work. If the gain vanishes under the stricter filter, the model was reading leaked results rather than learning to recognize causal claims.