Critical issues found in “Dissecting cell identity via network inference and in silico gene perturbation”

Jalil Nourisa, Antoine Passemiers, Sven TomfordebioRxiv 2024 · DOI

TL;DR

This paper critically re-evaluates CellOracle (CO), a recent method for inferring gene regulatory networks (GRNs) from single-cell RNA and ATAC-seq data. The authors identify significant limitations in CO's integration of ATAC-seq data, its benchmarking methodology, and the ground truth data used, suggesting that its claimed performance and ability to capture distal regulatory elements are questionable.

What Problem Does This Paper Address?

The original CellOracle (CO) paper, published in Nature, presented a novel method for constructing gene regulatory networks (GRNs) by integrating mRNA-seq and ATAC-seq data. GRNs are crucial for understanding how genes are regulated and how cellular identity is established. CO aimed to capture both proximal (promoter) and distal (enhancer) regulatory elements, using tools like Cicero to assess co-accessibility of genomic regions from ATAC-seq data. These GRNs were then intended for in silico gene perturbation studies to predict cellular behavior. However, this paper argues that the implementation and benchmarking of CO have critical flaws, potentially undermining its claimed capabilities and the validity of its results.

Methods

The authors meticulously dissected the CellOracle pipeline, focusing on two main areas:

  • ATAC-seq Data Integration: They analyzed how CO uses ATAC-seq data to identify regulatory elements. Specifically, they examined the get_tss_info and integrate_tss_peak_with_cicero functions. They found that CO's approach, by assigning an artificial co-accessibility score of 1 to promoter-TSS connections, effectively prioritizes these proximal interactions and fails to meaningfully incorporate distal regulatory elements identified by Cicero.
  • Benchmarking Methodology: The authors re-implemented CO's benchmarking process and identified a significant issue in how the all_combinations dataset was constructed. The original method included non-TF genes as potential regulatory genes, leading to a severe imbalance between positive and negative examples in the ground truth and predicted scores. This imbalance, they argue, compromises the reliability of the reported performance metrics. They propose an alternative method that exclusively uses confirmed transcription factors (TFs) from the ground truth data to construct all_combinations, aiming for a more balanced and accurate comparison.

Datasets

The study primarily uses publicly available datasets for benchmarking, including:

  • Mouse Tissue Data: Various mouse tissue samples (Heart, Kidney, Liver, Lung, Spleen) were used for ATAC-seq and mRNA-seq data.
  • ChIP-seq Data: Used as ground truth for GRN inference in the original CellOracle study.
  • Newly Introduced Benchmarks: The authors re-evaluated the original benchmarking approach and proposed a revised method for constructing the all_combinations dataset. They also highlight issues with the coverage and comprehensiveness of the ground truth data itself.

Key Insights and Ideas

  • Distal Elements Overlooked: CO's implementation, despite claiming to account for distal regulatory elements (enhancers), appears to heavily favor proximal promoter-TSS interactions due to how co-accessibility scores are processed and filtered.
  • Benchmarking Flaw: The original benchmarking approach suffers from a severe class imbalance due to the inclusion of non-TF genes in the potential regulatory gene set, rendering the reported performance metrics unreliable.
  • Ground Truth Limitations: The ground truth data used for benchmarking is found to be insufficient, with a significantly lower number of TFs than claimed, particularly for certain tissues, which can skew comparative analyses.
  • Regression Models' Limited Added Value: The regression models used to refine the base GRN from ATAC-seq data do not appear to extract significant additional regulatory insights from gene expression data, as their performance improvement over a shuffled GRN is minimal and statistically insignificant.

Key Results

  • The original CO pipeline, when analyzing ATAC-seq data, results in very sparse peak-gene connections, with a median of only 1 peak per gene.
  • When comparing GRN inference methods using the original benchmarking approach, CO yielded zero "Domain of Regulatory Chromatin" (DORC) scores, while other methods like Scenic+ and FigR yielded thousands and dozens, respectively.
  • After fixing the benchmarking methodology to address class imbalance, CO's performance remained comparable to other methods across various metrics (AUROC, EPR, AUPR, F1), but the overall scores for all methods dropped significantly, suggesting the original paper might have overestimated performance.
  • The ground truth data for benchmarking contained only 60 unique TFs across all tissues, with some tissues having as few as 2 TFs, contradicting the claim of 80 TFs and impacting the validity of comparisons.
  • The performance improvement of CO's regression models over its base GRN was minimal, with p-values of 0.4, 0.64, and 0.92 for AUROC, EPR, AUPR, and F1, respectively, indicating a lack of statistical significance.

Limitations and Future Work

  • Limitations:
    • CO's algorithm fails to adequately account for distal regulatory interactions.
  • The benchmarking algorithm and data used in the original study are flawed, leading to invalid comparative performance claims.
  • The ground truth data coverage is insufficient for robust benchmarking.
  • The added value of CO's regression models over its base GRN is minimal.
  • Future Work:
    • The authors advocate for a comprehensive re-evaluation of the CellOracle method and its published results to affirm its reliability for GRN inference and gene perturbation simulation.
  • Further investigation into how to meaningfully integrate distal regulatory elements with a more flexible threshold is suggested.
  • The study highlights the need for more comprehensive and balanced ground truth datasets for GRN benchmarking.

Summary updated

Abstract

1 Summary In the 2023 Nature publication “Dissecting cell identity via network inference and in silico gene perturbation” [1], the authors introduced CellOracle (CO), a novel method leveraging mRNA-seq and ATAC-seq data to construct gene regulatory networks (GRNs), which are subsequently used for gene perturbation. They designed CO to account for the role of distal cis-regulatory elements, e.g. enhancers, as well as proximal promoters in the gene regulation system. For this purpose, they employed Cicero to determine the co-accessibility scores between peaks, provided by ATAC-seq data. These scores are then used to identify the interaction of distal regions with the target gene. Using CO, they have conducted multiple perturbation studies on different organisms and identified novel phenotypes resulting from transcriptional factor (TF) perturbation. In addition, they benchmarked CO’s performance using ChIP-seq data as ground truth against other state-of-the-art GRN methods across multiple mouse tissue samples. However, our evaluation reveals critical limitations in the implementation of their methodology, both in terms of ATAC-seq data integration as well as benchmarking. In this report, we first explain the limitations in their approach of integrating ATAC-seq data. We show that the proposed algorithm fails to account for distal regulatory interactions. After, we present the issues associated with their benchmarking algorithm and the data used for benchmarking. We show that their findings regarding the comparative performance of CO against other GRN inference methods is invalid and requires further evaluation. In conclusion, we detect multiple inaccuracies in this paper which undermine the validity of their published protocol and the results. The materials supporting our findings are accessible on GitHub 1 .
BibTeX
@misc{nourisa2024critical,
  author = {Jalil Nourisa and Antoine Passemiers and Sven Tomforde},
  title = {{Critical issues found in “Dissecting cell identity via network inference and in silico gene perturbation”}},
  year = {2024},
  doi = {10.1101/2024.10.16.618746},
  howpublished = {bioRxiv},
}
Figure