TL;DR
This paper critically re-evaluates CellOracle (CO), a recent method for inferring gene regulatory networks (GRNs) from single-cell RNA and ATAC-seq data. The authors identify significant limitations in CO's integration of ATAC-seq data, its benchmarking methodology, and the ground truth data used, suggesting that its claimed performance and ability to capture distal regulatory elements are questionable.
What Problem Does This Paper Address?
The original CellOracle (CO) paper, published in Nature, presented a novel method for constructing gene regulatory networks (GRNs) by integrating mRNA-seq and ATAC-seq data. GRNs are crucial for understanding how genes are regulated and how cellular identity is established. CO aimed to capture both proximal (promoter) and distal (enhancer) regulatory elements, using tools like Cicero to assess co-accessibility of genomic regions from ATAC-seq data. These GRNs were then intended for in silico gene perturbation studies to predict cellular behavior. However, this paper argues that the implementation and benchmarking of CO have critical flaws, potentially undermining its claimed capabilities and the validity of its results.
Methods
The authors meticulously dissected the CellOracle pipeline, focusing on two main areas:
- ATAC-seq Data Integration: They analyzed how CO uses ATAC-seq data to identify regulatory elements. Specifically, they examined the
get_tss_infoandintegrate_tss_peak_with_cicerofunctions. They found that CO's approach, by assigning an artificial co-accessibility score of 1 to promoter-TSS connections, effectively prioritizes these proximal interactions and fails to meaningfully incorporate distal regulatory elements identified by Cicero. - Benchmarking Methodology: The authors re-implemented CO's benchmarking process and identified a significant issue in how the
all_combinationsdataset was constructed. The original method included non-TF genes as potential regulatory genes, leading to a severe imbalance between positive and negative examples in the ground truth and predicted scores. This imbalance, they argue, compromises the reliability of the reported performance metrics. They propose an alternative method that exclusively uses confirmed transcription factors (TFs) from the ground truth data to constructall_combinations, aiming for a more balanced and accurate comparison.
Datasets
The study primarily uses publicly available datasets for benchmarking, including:
- Mouse Tissue Data: Various mouse tissue samples (Heart, Kidney, Liver, Lung, Spleen) were used for ATAC-seq and mRNA-seq data.
- ChIP-seq Data: Used as ground truth for GRN inference in the original CellOracle study.
- Newly Introduced Benchmarks: The authors re-evaluated the original benchmarking approach and proposed a revised method for constructing the
all_combinationsdataset. They also highlight issues with the coverage and comprehensiveness of the ground truth data itself.
Key Insights and Ideas
- Distal Elements Overlooked: CO's implementation, despite claiming to account for distal regulatory elements (enhancers), appears to heavily favor proximal promoter-TSS interactions due to how co-accessibility scores are processed and filtered.
- Benchmarking Flaw: The original benchmarking approach suffers from a severe class imbalance due to the inclusion of non-TF genes in the potential regulatory gene set, rendering the reported performance metrics unreliable.
- Ground Truth Limitations: The ground truth data used for benchmarking is found to be insufficient, with a significantly lower number of TFs than claimed, particularly for certain tissues, which can skew comparative analyses.
- Regression Models' Limited Added Value: The regression models used to refine the base GRN from ATAC-seq data do not appear to extract significant additional regulatory insights from gene expression data, as their performance improvement over a shuffled GRN is minimal and statistically insignificant.
Key Results
- The original CO pipeline, when analyzing ATAC-seq data, results in very sparse peak-gene connections, with a median of only 1 peak per gene.
- When comparing GRN inference methods using the original benchmarking approach, CO yielded zero "Domain of Regulatory Chromatin" (DORC) scores, while other methods like Scenic+ and FigR yielded thousands and dozens, respectively.
- After fixing the benchmarking methodology to address class imbalance, CO's performance remained comparable to other methods across various metrics (AUROC, EPR, AUPR, F1), but the overall scores for all methods dropped significantly, suggesting the original paper might have overestimated performance.
- The ground truth data for benchmarking contained only 60 unique TFs across all tissues, with some tissues having as few as 2 TFs, contradicting the claim of 80 TFs and impacting the validity of comparisons.
- The performance improvement of CO's regression models over its base GRN was minimal, with p-values of 0.4, 0.64, and 0.92 for AUROC, EPR, AUPR, and F1, respectively, indicating a lack of statistical significance.
Limitations and Future Work
- Limitations:
- CO's algorithm fails to adequately account for distal regulatory interactions.
- The benchmarking algorithm and data used in the original study are flawed, leading to invalid comparative performance claims.
- The ground truth data coverage is insufficient for robust benchmarking.
- The added value of CO's regression models over its base GRN is minimal.
- Future Work:
- The authors advocate for a comprehensive re-evaluation of the CellOracle method and its published results to affirm its reliability for GRN inference and gene perturbation simulation.
- Further investigation into how to meaningfully integrate distal regulatory elements with a more flexible threshold is suggested.
- The study highlights the need for more comprehensive and balanced ground truth datasets for GRN benchmarking.
Summary updated