Incorporating Large Language Model-Derived Information into Hypothesis Testing for Genomics

Jordan Bryan, Hui Niu, Didong LibioRxiv 2025 · DOI

Abstract

We propose strategies for incorporating the information in large language models (LLMs) into statistical hypothesis tests in genomics studies. Using gene embeddings derived from text inputs to OpenAI's GPT-3.5 model, we show that biological signals in a variety of genomics datasets reside near the principal subspace spanned by the embeddings. We then use a frequentist and Bayesian (FAB) framework to propose three hypothesis tests that are optimal with respect to prior information based on the gene embedding subspace. In three separate real-world genomics examples, the FAB tests guided by the LLM-derived information achieve more power than classical counterparts.

Introduction

In modern genomics, researchers frequently encounter a "needle in a haystack" problem. Testing thousands of genes or perturbations simultaneously often results in low statistical power, making it difficult to detect subtle but biologically significant signals. Traditional methods like $t$-tests or $F$-tests typically assume that signals are equally likely to appear in any direction within the data space. However, decades of biological research suggest that certain pathways and gene relationships are more likely to be involved in specific phenotypes than others.

This research introduces a novel approach to bridge the gap between unstructured biological knowledge and formal statistical testing. By leveraging Large Language Models (LLMs) to extract information from scientific literature, we can "prime" statistical tests to look in the right places without compromising the mathematical rigor required for error control.

Methodology: The FAB Framework

The core of this approach is the Frequentist and Bayesian (FAB) framework. This hybrid method utilizes a Bayesian prior to focus the test's power on specific areas of the data space where a signal is expected, based on LLM-derived insights. Crucially, it remains Frequentist in its execution: it guarantees that the Type I error rate (false positive rate) is maintained at a pre-specified level (e.g., $\alpha = 0.05$), regardless of whether the LLM's prior information is accurate.

The workflow consists of three main components:

  1. Gene Embeddings: Utilizing GenePT, which uses OpenAI’s GPT-3.5 to convert NCBI text descriptions of genes into high-dimensional numerical vectors.
  2. Subspace Projection: Identifying the principal components of these embeddings to define a "biological subspace."
  3. Optimized Test Statistics: Developing three specific tests: AFAB (Approximate FAB) for comparing high-dimensional means, the FAB $F$-test for linear hypotheses, and the FAB $t$-test for simultaneous gene-level testing.

The Biological Subspace Hypothesis

A key discovery of this research is that biological signals—such as the differential expression between leukemia types—tend to align with the principal components of LLM embeddings. This suggests that LLMs have effectively "learned" a low-dimensional representation of functional biology from the scientific literature.

Fraction of total signal norm captured by LLM embeddings versus random projections.

As shown in the figure above, the LLM-derived embeddings (blue line) capture a significantly higher fraction of the total signal norm in lower dimensions compared to random projections (gray lines). This concentration of signal allows the FAB tests to achieve higher power by focusing on these informative dimensions.

Experimental Evaluation and Results

The researchers validated these methods across several high-profile genomics datasets, including DepMap, CRISPRi Perturb-seq, and the PRISM drug screen.

  • Increased Discovery Power: In the CRISPRi Perturb-seq analysis, the AFAB test identified significantly more bioactive perturbations than standard randomized projection tests.
  • Drug Sensitivity: In the PRISM drug screen, the FAB $F$-test identified 193 drugs with significant viability patterns at a 0.1 False Discovery Rate (FDR), a nearly tenfold increase over the 20 drugs found by the classical $F$-test.
  • Identifying "Known Unknowns": In an AML CRISPR screen, the FAB $t$-test successfully identified KRAS as a significant dependency, along with GNG5 and FURIN. While these genes have known links to Ras pathways, they were missed by classical $t$-tests and only reached significance with the LLM-guided "nudge."

Discussion and Future Directions

The FAB framework provides a "safety net" for researchers: if the LLM embeddings are informative, the test gains substantial power; if they are irrelevant, the test performs similarly to standard methods without inflating false positives.

However, several areas for future work remain. The current models primarily exploit linear relationships; incorporating non-linear functions of embeddings could further enhance flexibility. Additionally, the "black box" nature of embeddings makes it difficult to pinpoint which specific textual information drives the statistical power. Future research will also explore more advanced models like GPT-4 or biology-specific LLMs like BioBERT to further refine the prior information used in these tests.

Summary updated

Abstract

We propose strategies for incorporating the information in large language models (LLMs) into statistical hypothesis tests in genomics studies. Using gene embeddings derived from text inputs to OpenAI's GPT-3.5 model, we show that biological signals in a variety of genomics datasets reside near the principal subspace spanned by the embeddings. We then use a frequentist and Bayesian (FAB) framework to propose three hypothesis tests that are optimal with respect to prior information based on the gene embedding subspace. In three separate real-world genomics examples, the FAB tests guided by the LLM-derived information achieve more power than classical counterparts.
BibTeX
@misc{bryan2025incorporating,
  author = {Jordan Bryan and Hui Niu and Didong Li},
  title = {{Incorporating Large Language Model-Derived Information into Hypothesis Testing for Genomics}},
  year = {2025},
  doi = {10.1101/2025.04.30.651464},
  howpublished = {bioRxiv},
}
Figure