Heterogeneous gene network estimation for single-cell transcriptomic data via a joint regularized deep neural network

Jingyuan Yang, Tao Li, Tianyi Wang, Shuangge Ma, Mengyun WuJournal of the American Statistical Association 2026 · DOI

TL;DR

This paper introduces JRDNN-KM, a novel deep learning framework that simultaneously identifies cell subgroups and reconstructs gene regulatory networks from single-cell transcriptomic data. By effectively handling cellular heterogeneity and complex nonlinear relationships, JRDNN-KM offers more accurate and biologically interpretable insights into cellular processes compared to existing methods.

What Problem Does This Paper Address?

Understanding how genes interact within cells (gene networks) is crucial for deciphering biological processes and disease mechanisms. Single-cell transcriptomic data offers an unprecedented view into cellular diversity, but analyzing it is challenging due to inherent heterogeneity, technical noise (like "dropout" events where gene expression is not detected), and the complex, often nonlinear, relationships between genes. Existing methods often struggle with these challenges, either by oversimplifying gene interactions (linear models) or by not adequately accounting for the distinct characteristics of different cell types within a sample. This paper aims to bridge this gap by developing a method that can simultaneously identify these distinct cell groups and map their unique gene interaction networks.

Methods

The core of the proposed method is the Joint Regularized Deep Neural Network with K-Means Clustering (JRDNN-KM).

  • JRDNN (Joint Regularized Deep Neural Network): This is a deep neural network designed to learn gene networks. It incorporates:
  • Heterogeneous and Homogeneous Neurons: Hidden layers include neurons specific to each cell subgroup (heterogeneous) and neurons shared across all subgroups (homogeneous). This allows the model to capture both common patterns and subgroup-specific variations.
  • Selection Layer: A specialized layer helps in constructing sparse networks by identifying important gene-gene connections.
  • Zero-Inflated Loss Function: This function is tailored to handle the "dropout" events common in single-cell data, ensuring that the model is robust to missing expression values.
  • Regularization Terms: L1 and other regularization techniques are used to promote sparsity in the network connections and encourage similarity between networks of different subgroups.
  • Mahalanobis Distance-based K-Means Clustering (KM): This clustering algorithm is used to identify cell subgroups. Crucially, it uses the precision matrices (derived from the JRDNN-estimated networks) to calculate distances between cells, allowing it to cluster cells based on their gene network similarity, not just their expression levels.

The JRDNN-KM method operates iteratively: it first uses the JRDNN to estimate gene networks based on current subgroup assignments, then uses these networks to refine the cell subgroup assignments via K-means clustering, and repeats until convergence.

Datasets

The study evaluates JRDNN-KM on five real single-cell transcriptomic datasets:

  • Human lung adenocarcinoma (LUAD) cell line.
  • Human peripheral blood mononuclear cells (PBMC).
  • Mouse embryonic stem cells (mESCs).
  • Mouse liver tissues.
  • Mouse uterus tissues.

These datasets were chosen for their diversity in tissue type, developmental stage, and physiological context, as well as variations in sample size, data quality (e.g., dropout rates), and cell type composition.

Key Insights and Ideas

  • Simultaneous Subgroup and Network Inference: The most significant insight is the power of jointly learning cell subgroups and their corresponding gene networks. This approach allows the network structure to inform cell clustering and vice-versa, leading to more accurate and biologically meaningful results.
  • Capturing Nonlinearity is Crucial: The paper strongly emphasizes that gene interactions are often nonlinear, and methods that only consider linear relationships miss important biological insights. JRDNN-KM's deep learning architecture excels at capturing these complex nonlinear dependencies.
  • Heterogeneity and Homogeneity are Key: By incorporating both subgroup-specific and shared neurons, the model can disentangle what's unique to certain cell types from what's common across different cell types within a tissue, providing a more nuanced understanding.
  • Network Structure as a Basis for Clustering: Using gene network similarity (via precision matrices) for clustering is a novel approach that goes beyond traditional expression-based clustering, revealing subgroups that might be missed otherwise.

Key Results

  • Superior Cell Subgroup Identification: JRDNN-KM achieved high accuracy (ARI and NMI values close to 1.00) in identifying cell subgroups across multiple datasets, often outperforming existing state-of-the-art methods like SC3 and Seurat.
  • Accurate Network Reconstruction: The method demonstrated high F1 scores when comparing its inferred networks against established biological databases (CORUM, STRING, PerturbAtlas), indicating its ability to reconstruct biologically relevant gene interactions.
  • Identification of Biologically Relevant Genes: JRDNN-KM successfully identified known marker genes and also revealed novel potential biomarkers and biological pathways (e.g., related to ferroptosis and protein modification in LUAD), many of which were overlooked by other methods.
  • Robustness: The method showed consistent performance across diverse datasets, including those with high dropout rates and imbalanced subgroup distributions, and maintained stability under subsampling analysis.

Limitations and Future Work

  • Loss Function Robustness: The authors acknowledge that the proposed loss function, based on squared loss, might lack robustness against outliers and noise. Future work could explore optimizing this aspect.
  • Zero Inflation Modeling: While the method uses zero-inflated models, emerging research suggests technical dropout might be correlated with expression levels. A more nuanced approach to account for these dependencies is suggested.
  • Estimating Effect Magnitudes: Similar to many deep learning models, precisely estimating the strength of nonlinear relationships remains challenging and is deferred to future study.
  • Computational Efficiency: For very large numbers of genes (p), computational complexity can be an issue. Future work will focus on optimizing the algorithm for better scalability.
  • Alternative Distributions: Exploring alternative distributions like Poisson or negative binomial for zero-inflated models, especially with deep neural networks, is a promising avenue for methodological innovation.

Summary updated

BibTeX
@article{yang2026heterogeneous,
  author = {Jingyuan Yang and Tao Li and Tianyi Wang and Shuangge Ma and Mengyun Wu},
  title = {{Heterogeneous gene network estimation for single-cell transcriptomic data via a joint regularized deep neural network}},
  journal = {Journal of the American Statistical Association},
  year = {2026},
  doi = {10.1080/01621459.2026.2615185},
}
Figure