Nonparametric clustering of <scp>RNA</scp>‐sequencing data

Gabriel Lozano, Nadia A. Lanman, Michael Levine

TL;DR

This paper introduces a novel nonparametric clustering algorithm, npMSL, for RNA-sequencing data. By avoiding assumptions about specific probability distributions for gene expression, npMSL offers greater interpretability and can uncover biologically meaningful gene clusters that parametric methods might miss.

What Problem Does This Paper Address?

RNA-sequencing (RNA-seq) data is crucial for understanding gene expression and its role in biological processes. A key task is identifying groups of genes that are "co-expressed," meaning their expression levels tend to rise and fall together. This co-expression can reveal genes involved in similar biological functions or pathways.

Existing methods for this task generally fall into two categories: distance-based and model-based. While model-based approaches offer better interpretability by defining clusters through explicit models, they often struggle with a major challenge: selecting the correct multivariate probability distribution to model the complex, often skewed, and overdispersed nature of RNA-seq data. This difficulty in choosing a distribution can hinder the effectiveness and interpretability of the clustering results.

Methods

The core contribution of this paper is the application of a nonparametric Maximum Smoothed Likelihood (npMSL) algorithm to RNA-seq data clustering. Unlike traditional model-based methods that assume specific parametric distributions (like Poisson or negative binomial), npMSL takes a more flexible approach.

Here's a breakdown of the key methodological aspects:

  • Nonparametric Mixture Model: The algorithm models gene expression data as a mixture of clusters, where each cluster's density function is not assumed to belong to any predefined parametric family. This is a significant departure from standard practice.
  • Conditionally Independent Marginals: A key assumption is that within each cluster, the expression levels of different genes are independent, conditional on belonging to that cluster. This simplifies the model while still allowing for complex relationships.
  • Maximization-Minorization (MM) Algorithm: The npMSL algorithm is an MM algorithm, which is guaranteed to converge to a local optimum. It iteratively refines estimates of cluster parameters (like mixing proportions and density functions) by maximizing a smoothed log-likelihood function.
  • Bandwidth and Kernel Selection: The method uses kernel density estimation, and the paper discusses the selection of bandwidth (a smoothing parameter). They opt for a practical approach using Silverman's rule of thumb, acknowledging it's a simplification.
  • Model Selection for Number of Clusters: To determine the optimal number of clusters, the authors fit Gaussian repeated measures models over a range of cluster numbers and use common model selection criteria (AIC, BIC, CAIC, ICL) to guide the choice.

Datasets

The paper evaluates the npMSL algorithm on two real-world RNA-seq datasets:

  1. Mouse Tissue Dataset: This dataset comprises bulk RNA-seq data from lung, kidney, liver, and small intestine tissues of mice. It includes two biological replicates per tissue type. The data is available through the Gene Expression Omnibus (GEO) repository under accession number GSE124419.
  2. Human Prostate Cancer Cell Line Dataset: This dataset includes RNA-seq data from four human prostate cancer cell lines, with three replicates each. These cell lines represent both chemotherapy-sensitive and resistant states, making them relevant for studying cancer biology.

The authors do not introduce a new benchmark dataset but rather apply their method to existing, well-characterized datasets to demonstrate its utility and compare it with established methods.

Key Insights and Ideas

  • Breaking Free from Parametric Constraints: The most significant insight is that by avoiding rigid distributional assumptions, npMSL can capture more nuanced patterns in RNA-seq data. This is crucial because real biological data rarely perfectly fits theoretical distributions.
  • Enhanced Interpretability: Because each cluster is characterized by a flexible, estimated density function rather than a fixed parametric form, the biological meaning of each cluster can be more directly understood.
  • Uncovering Hidden Clusters: The npMSL method demonstrated the ability to identify biologically meaningful clusters that might be missed by traditional parametric approaches, suggesting it can reveal novel biological insights.
  • Robustness to Data Characteristics: The nonparametric nature makes the algorithm inherently more robust to the skewed and overdispersed nature of RNA-seq data, which often poses challenges for parametric models.

Key Results

  • Biologically Meaningful Clusters: On both the mouse tissue and prostate cancer cell line datasets, the npMSL algorithm successfully identified a substantial number of biologically relevant gene clusters.
  • Performance Comparison: The npMSL method was compared to two other common RNA-seq clustering methods: a Poisson mixture model and a transformation-based method.
  • For the mouse tissue dataset, npMSL identified 17 clusters, with 94.1% showing significant enrichment of biological processes.
  • For the prostate cancer cell line dataset, npMSL identified 17 clusters, with 82.4% showing significant GO term enrichment. In this case, npMSL outperformed the other methods in terms of the proportion of enriched clusters.
  • Divergent Solutions: Adjusted Rand Index (ARI) values indicated that the clustering solutions produced by npMSL were significantly different from those generated by the Poisson and transformation-based methods, suggesting that npMSL is indeed uncovering distinct patterns.
  • Code Availability: The authors have made their code publicly available on GitHub, facilitating reproducibility and adoption by other researchers.

Limitations and Future Work

  • Bandwidth Selection: The paper acknowledges that the selection of the bandwidth parameter for kernel density estimation is a challenging problem. While they use a practical rule of thumb, they note that it's a crude method and could lead to under- or oversmoothing. They suggest that a more sophisticated, perhaps iterative, approach to bandwidth selection would be beneficial, though it might violate the monotonicity property of the algorithm.
  • Conditional Independence Assumption: The current npMSL model assumes conditional independence of biological samples within a cluster. The authors recognize that this might be a limitation, especially for datasets with inherent dependencies between samples.
  • Future Directions:
    • Developing more advanced methods for bandwidth selection in the context of nonparametric mixture models.
  • Extending the model to account for potential dependence structures between biological samples, possibly by incorporating copula functions. This would represent a significant step towards a more comprehensive nonparametric modeling framework.

Summary updated

Abstract

Abstract Identification of clusters of co‐expressed genes in transcriptomic data is a difficult task. Most algorithms used for this purpose can be classified into two broad categories: distance‐based or model‐based approaches. Distance‐based approaches typically utilize a distance function between pairs of data objects and group similar objects together into clusters. Model‐based approaches are based on using the mixture‐modeling framework. Compared to distance‐based approaches, model‐based approaches offer better interpretability because each cluster can be explicitly characterized in terms of the proposed model. However, these models present a particular difficulty in identifying a correct multivariate distribution that a mixture can be based upon. In this manuscript, we review some of the approaches used to select a distribution for the needed mixture model first. Then, we propose avoiding this problem altogether by using a nonparametric MSL (maximum smoothed likelihood) algorithm. This algorithm was proposed earlier in statistical literature but has not been, to the best of our knowledge, applied to transcriptomics data. The salient feature of this approach is that it avoids explicit specification of distributions of individual biological samples altogether, thus making the task of a practitioner easier. We performed both a simulation study and an application of the proposed algorithm to two different real datasets. When used on a real dataset, the algorithm produces a large number of biologically meaningful clusters and performs at least as well as several other mixture‐based algorithms commonly used for RNA‐seq data clustering. Our results also show that this algorithm is capable of uncovering clustering solutions that may go unnoticed by several other model‐based clustering algorithms. Our code is publicly available on Github at https://github.com/Matematikoi/non_parametric_clustering
BibTeX
@inproceedings{lozano2023nonparametric,
  author = {Gabriel Lozano and Nadia A. Lanman and Michael Levine},
  title = {{Nonparametric clustering of <scp>RNA</scp>‐sequencing data}},
  booktitle = {Statistical},
  year = {2023},
  doi = {10.1002/sam.11638},
}
Figure