Abstract
Modern cell-perturbation experiments expose cells to panels of hundreds of stimuli, such as cytokines or CRISPR guides that perform gene knockouts. These experiments are designed to investigate whether a particular gene is upregulated or downregulated by exposure to each treatment. However, due to high levels of experimental noise, typical estimators of whether a gene is up- or down-regulated make many errors. In this paper, we make two contributions. Our first contribution is a new estimator of regulatory effect that makes use of Gaussian processes and factor analysis to leverage auxiliary information about similarities among treatments, such as the chemical similarity among the drugs used to perturb cells. The new estimator typically has lower variance than unregularized estimators, which do not use auxiliary information, but higher bias. To assess whether this new estimator improves accuracy (i.e., achieves a favorable trade-off between bias and variance), we cannot simply compute its error on heldout data as "ground truth" about the effects of treatments is unavailable. Our second contribution is a novel data-splitting method to evaluate error rates. This data-splitting method produces valid error bounds using "sign-valid" estimators, which by definition have the correct sign more often than not. Using this data-splitting method, through a series of case studies we find that our new estimator, which leverages auxiliary information, can yield a three-fold reduction in type S error rate.
Introduction
In modern biology, high-throughput experiments are essential for understanding how thousands of perturbations—such as drugs or CRISPR gene knockouts—affect gene expression. The primary goal is to determine the Average Treatment Effect (ATE): does a specific treatment turn a gene "up" or "down"?
However, these experiments are plagued by extreme noise. This noise often leads to "Type S" (sign) errors, where the estimated ATE has the opposite sign of the true biological effect. Misinterpreting a suppressor as an activator can lead to fundamental misunderstandings of biological control circuits. Furthermore, the lack of "ground truth" in these experiments makes it difficult to measure error rates or validate new methodologies designed to mitigate them.
The Smoothed Estimator
To address these challenges, we propose a "smoothed" estimator. Unlike standard approaches that treat every treatment-gene pair as an isolated measurement, our model uses Gaussian Processes (GPs) and factor analysis to incorporate auxiliary "side information."
For instance, if two drugs share similar chemical structures, the model assumes they likely exert similar biological effects. By employing a Kronecker-structured covariance matrix, the model shares information across similar treatments and correlated genes. This effectively "borrows strength" from related experiments to smooth out experimental noise. The parameters for this model are optimized using an Expectation-Maximization (EM) procedure.
Evaluating Accuracy Without Ground Truth
Our second contribution is a novel data-splitting evaluation method. By splitting experimental replicates into two groups, we use a "sign-valid" estimator (a simple, unregularized one) on one half to calculate the Cross-replicate Sign Proportion (CSEP). We mathematically prove that CSEP can serve as a valid upper bound on the true Type S error rate of the smoothed estimator, even in the absence of ground truth data.
Results and Key Insights
We validated our approach using three real-world datasets (CRISPRa, L1000, and Sci-Plex) and semi-synthetic benchmarks. Our findings demonstrate that the smoothed estimator significantly outperforms standard "raw" estimators.
As shown in the figure above, the smoothed estimator (red) consistently maintains a lower CSEP than the raw estimator (blue) across various T-cell conditions. Key results include:
- Three-fold Error Reduction: The smoothed estimator yielded a three-fold reduction in the Type S error rate in several case studies.
- Increased Discovery Power: On the Sci-Plex and L1000 datasets, the smoothed estimator allowed for 2.5x to 3.5x more high-confidence discoveries at the same error-rate threshold.
- Biological Insight: The model's learned hyperparameters quantified experimental design factors, such as the relative similarity between CRISPR guides versus different stimulation states.
While smoothing introduces bias, our results confirm that the massive reduction in variance leads to a much more accurate identification of biological signals.
Discussion and Future Work
While effective, Gaussian Processes typically scale cubically ($O(N^3)$) with data size. Future work will focus on scaling these methods to "atlas-scale" datasets using advanced inference techniques like those in GPyTorch. Additionally, we aim to address "estimand mismatch" when integrating data across different platforms (e.g., bulk vs. single-cell RNA sequencing) and explore weighted error metrics that prioritize key genes of interest.
Summary updated