Large Language Models for Zero-shot Inference of Causal Structures in Biology

Izzy Newsham, Luka Kovačević, Richard Moulange, Nan Rosemary Ke, Sach Mukherjee

TL;DR

This paper investigates whether large language models (LLMs) can infer directed causal relationships between genes in a completely zero-shot setting—without seeing experimental data beforehand. By benchmarking predictions against real-world CRISPR intervention screens (Perturb-seq), the authors find that even a modest 9B-parameter LLM beats established interaction databases (like STRING). Crucially, while specifying experimental context (cell type and measurement modality) improves performance, feeding the model gene-specific literature excerpts actually degrades it.

What Problem Does This Paper Address?

Understanding gene regulatory networks—how perturbing one gene causally impacts others—is central to disease research and drug discovery. However, the space of possible gene interactions is astronomically large, making exhaustive lab experiments impossible.

While LLMs are increasingly deployed as scientific assistants, their ability to infer true directed causal structure in biological systems remains unproven. Classical biomedical databases and text-mining pipelines capture associations or undirected protein interactions, but they often struggle to distinguish causal direction or context-specific effects (e.g., tissue- or disease-specific behavior). This paper asks: Can an off-the-shelf LLM predict whether Gene A causally affects Gene B in a specific cellular context without prior task-specific fine-tuning?

Methods

The authors designed a systematic benchmarking framework:

Causal Ground Truth Construction: Using Perturb-seq data, the authors applied the Mann–Whitney $U$ test with Benjamini–Hochberg false-discovery rate correction ($\alpha = 0.05$) to compare single-cell gene expression distributions between perturbed (CRISPRi knockdown) and unperturbed control cells. A significant shift defines a directed causal edge $k \rightarrow j$ (capturing ancestral/indirect effects).
Pairwise LLM Querying: Using Gemma2-9B-it, the model was queried across all $d(d - 1) = 9,900$ pairs of genes to output a continuous probability $p_{i \to j} \in [0, 1]$ indicating the strength of a causal effect. The model was reinstantiated after every prompt to prevent multi-turn bias.
Prompt & Context Engineering: The team tested 105 prompt variations manipulating:
Experimental context: specifying disease/cell line (human CML leukemia), readout type (mRNA expression), and protocol details.
Gene-level evidence: raw RefSeq gene descriptions or PubTator 3.0 literature passages.
Reasoning frameworks: testing simple and guided Chain-of-Thought (CoT) reasoning.
Evaluation: Evaluated edge prediction using the Area Under the Receiver Operating Characteristic curve (AUROC) against the experimental causal graph (and its transitive closure).

Datasets

Perturb-seq Screen: Data from Replogle et al. (2022) consisting of $>2.5$ million single human cells. The authors evaluated a sub-network of the top 100 cancer-relevant genes perturbed and measured in the K562 chronic myelogenous leukemia (CML) cell line.
Knowledge Baselines & Retrieval Corpora:
RefSeq: Functional gene summaries.
PubTator 3.0 API: Mined abstracts containing relation types such as associate, interact, positive correlate, and negative correlate.
STRING Database: A state-of-the-art protein-protein interaction database used as a classical knowledge-driven baseline.

Key Insights and Ideas

Experimental Context Helps; Gene Literature Hurts: Providing high-level experimental context (e.g., stating that measurements reflect mRNA levels in human CML cells) boosted prediction accuracy. However, directly feeding literature abstracts or gene descriptions consistently lowered performance.
The "Literature Trap" in Causal Biology: A statistical test between the ground-truth causal graph and PubTator literature mentions showed no significant correlation (Boschloo’s exact test, $p = 0.2075$). Biomedical literature is heavily biased toward studied phenotypes rather than systemic causal outcomes. When prompted with gene descriptions, the LLM over-indexed on functional differences (e.g., concluding two genes cannot interact because one regulates DNA repair and the other mediates cell adhesion).
Chain-of-Thought (CoT) Reduces Confidence: Counterintuitively, applying CoT degraded AUROC. Guided step-by-step reasoning encouraged the model to over-deliberate on unobserved confounding factors, making it overly conservative and flattening predictive probability spreads.
LLMs Outperform Classical Knowledge Networks for Causality: The LLM's zero-shot predictions substantially beat the STRING database baseline, which relies on curated associations and symmetric interaction scores rather than directed causality.

Key Results

Peak Performance: The best prompt configuration (cancer + mRNA experimental context without gene-specific text) achieved an AUROC of 0.625 ($\pm 0.0016$).
Baseline Comparison: The STRING database scored an AUROC of 0.460 on an undirected version of the ground-truth graph, confirming that standard association databases perform worse than random chance when tasked with directed causal screening.
Directional Sensitivity: In a case study on the gene pair ATR $\rightarrow$ CD47:
Naive prompt: $p_{\text{ATR}\to\text{CD47}} = 0.25$ vs. $p_{\text{CD47}\to\text{ATR}} = 0.15$.
With cancer context: $p_{\text{ATR}\to\text{CD47}} = 0.35$ vs. $p_{\text{CD47}\to\text{ATR}} = 0.05$.
With contradictory literature: The model successfully identified the correct causal direction ($p=0.75$ vs. $p=0.0$).

Limitations and Future Work

Small Model Architecture: Experiments were restricted to Gemma2-9B-it due to compute constraints (large context experiments took up to 32 hours per run on an A100 GPU). Larger frontier models or reasoning-focused models (e.g., OpenAI o1) were not evaluated.
Modest Absolute Accuracy: While an AUROC of 0.625 clearly beats random guessing and traditional databases, it is still modest. Its immediate utility lies in serving as an informative prior to constrain data-driven causal structure learning algorithms rather than as a standalone oracle.
Context-Specific RAG: Future work should explore more surgical retrieval-augmented generation (RAG) pipelines that retrieve literature strictly matching the specific cell type and perturbation modality, filtering out general or mismatched biological contexts.

Loading related content...