Improving accuracy in cell-perturbation experiments by leveraging auxiliary information

Jackson Loper, Noam Solomon, Jeffrey RegierThe Annals of Applied Statistics 2023 · arXiv

Abstract

Modern cell-perturbation experiments expose cells to panels of hundreds of stimuli, such as cytokines or CRISPR guides that perform gene knockouts. These experiments are designed to investigate whether a particular gene is upregulated or downregulated by exposure to each treatment. However, due to high levels of experimental noise, typical estimators of whether a gene is up- or down-regulated make many errors. In this paper, we make two contributions. Our first contribution is a new estimator of regulatory effect that makes use of Gaussian processes and factor analysis to leverage auxiliary information about similarities among treatments, such as the chemical similarity among the drugs used to perturb cells. The new estimator typically has lower variance than unregularized estimators, which do not use auxiliary information, but higher bias. To assess whether this new estimator improves accuracy (i.e., achieves a favorable trade-off between bias and variance), we cannot simply compute its error on heldout data as "ground truth" about the effects of treatments is unavailable. Our second contribution is a novel data-splitting method to evaluate error rates. This data-splitting method produces valid error bounds using "sign-valid" estimators, which by definition have the correct sign more often than not. Using this data-splitting method, through a series of case studies we find that our new estimator, which leverages auxiliary information, can yield a three-fold reduction in type S error rate.

Introduction

In modern biology, high-throughput experiments are essential for understanding how thousands of perturbations—such as drugs or CRISPR gene knockouts—affect gene expression. The primary goal is to determine the Average Treatment Effect (ATE): does a specific treatment turn a gene "up" or "down"?

However, these experiments are plagued by extreme noise. This noise often leads to "Type S" (sign) errors, where the estimated ATE has the opposite sign of the true biological effect. Misinterpreting a suppressor as an activator can lead to fundamental misunderstandings of biological control circuits. Furthermore, the lack of "ground truth" in these experiments makes it difficult to measure error rates or validate new methodologies designed to mitigate them.

The Smoothed Estimator

To address these challenges, we propose a "smoothed" estimator. Unlike standard approaches that treat every treatment-gene pair as an isolated measurement, our model uses Gaussian Processes (GPs) and factor analysis to incorporate auxiliary "side information."

For instance, if two drugs share similar chemical structures, the model assumes they likely exert similar biological effects. By employing a Kronecker-structured covariance matrix, the model shares information across similar treatments and correlated genes. This effectively "borrows strength" from related experiments to smooth out experimental noise. The parameters for this model are optimized using an Expectation-Maximization (EM) procedure.

Expectation-Maximization algorithm for fitting the smoothed estimator's parameters.

Evaluating Accuracy Without Ground Truth

Our second contribution is a novel data-splitting evaluation method. By splitting experimental replicates into two groups, we use a "sign-valid" estimator (a simple, unregularized one) on one half to calculate the Cross-replicate Sign Proportion (CSEP). We mathematically prove that CSEP can serve as a valid upper bound on the true Type S error rate of the smoothed estimator, even in the absence of ground truth data.

Results and Key Insights

We validated our approach using three real-world datasets (CRISPRa, L1000, and Sci-Plex) and semi-synthetic benchmarks. Our findings demonstrate that the smoothed estimator significantly outperforms standard "raw" estimators.

Comparison of CSEP (an upper bound on Type S error) between raw and smoothed estimators across CD4 and CD8 T-cell conditions.

As shown in the figure above, the smoothed estimator (red) consistently maintains a lower CSEP than the raw estimator (blue) across various T-cell conditions. Key results include:

  • Three-fold Error Reduction: The smoothed estimator yielded a three-fold reduction in the Type S error rate in several case studies.
  • Increased Discovery Power: On the Sci-Plex and L1000 datasets, the smoothed estimator allowed for 2.5x to 3.5x more high-confidence discoveries at the same error-rate threshold.
  • Biological Insight: The model's learned hyperparameters quantified experimental design factors, such as the relative similarity between CRISPR guides versus different stimulation states.

While smoothing introduces bias, our results confirm that the massive reduction in variance leads to a much more accurate identification of biological signals.

Discussion and Future Work

While effective, Gaussian Processes typically scale cubically ($O(N^3)$) with data size. Future work will focus on scaling these methods to "atlas-scale" datasets using advanced inference techniques like those in GPyTorch. Additionally, we aim to address "estimand mismatch" when integrating data across different platforms (e.g., bulk vs. single-cell RNA sequencing) and explore weighted error metrics that prioritize key genes of interest.

Summary updated

Abstract

Modern cell-perturbation experiments expose cells to panels of hundreds of treatments, such as cytokines or CRISPR guides. These experiments are designed to investigate whether a particular gene is up- or downregulated by exposure to each treatment. Typical estimators of whether a gene is up- or down-regulated make many errors due to high levels of experimental noise and complex donor and batch effects. In this article we explore whether estimator error rates can be reduced by pooling measurements according to auxiliary information, such as prior knowledge of treatments that are likely to have similar effects. Because auxiliary information can be misleading and because it is generally infeasible to model its quality, we take a model-free approach to identifying a subset of estimates in which error is reliably controlled. We validate this overall approach in simulation, comparing it to an alternative based on the Benjamini–Hochberg procedure. We then use the proposed approach to reanalyze a high-throughput CRISPRa screening and a small-molecule drug screening, which leads to new discoveries about the effects of gene augmentation on immune cells and the effects of cytokines on cancer cells.
BibTeX
@article{loper2023improving,
  author = {Jackson Loper and Noam Solomon and Jeffrey Regier},
  title = {{Improving accuracy in cell-perturbation experiments by leveraging auxiliary information}},
  journal = {The Annals of Applied Statistics},
  year = {2023},
}
Figure