Nonparametric clustering of RNA-sequencing data

Gabriel Lozano, Nadia A. Lanman, Michael Levine

TL;DR

This paper introduces a nonparametric approach to clustering RNA-sequencing data using the Maximum Smoothed Likelihood (npMSL) algorithm. By moving away from rigid parametric assumptions (like Poisson or Negative Binomial distributions), the authors provide a flexible framework that identifies biologically meaningful gene clusters more effectively than traditional mixture models.

What Problem Does This Paper Address?

Clustering genes with similar expression patterns is a cornerstone of transcriptomics, helping researchers identify co-regulated pathways and functional modules. However, RNA-seq data is notoriously difficult to model: it is highly skewed, exhibits a massive dynamic range, and is almost always "overdispersed" (the variance is much higher than the mean).

Current model-based clustering methods usually rely on finite mixture models. The catch? You have to pre-select a specific probability distribution (like a Multivariate Poisson-lognormal) for the clusters. If your chosen distribution doesn't perfectly match the messy reality of biological data, the resulting clusters can be misleading. There are very few reliable "goodness-of-fit" tests for these complex distributions, leaving practitioners to guess which model to use.

Methods

The authors propose using the npMSL (nonparametric Maximum Smoothed Likelihood) algorithm. While this algorithm has existed in statistical literature, this is its first major application to transcriptomics.

Key features of the approach: * Distribution Agnostic: Instead of assuming a cluster follows a specific family (like Gaussian or Poisson), it treats the cluster densities as "just" probability density functions to be estimated from the data. * Conditional Independence: It assumes that, given a specific cluster, the measurements across different conditions/samples are independent. This simplifies the math while remaining a standard assumption in many RNA-seq tools. * The MM Algorithm: To solve the optimization problem, they use a Maximization-Minorization (MM) algorithm. Much like the more famous EM algorithm, it is guaranteed to converge and possesses a "monotonicity property" (it gets better with every step). * Kernel Smoothing: The method uses a kernel density function and a bandwidth (selected via Silverman’s rule of thumb) to "smooth" the likelihood, preventing the model from over-fitting to noise.

Datasets

The authors evaluated their method on two real-world datasets:

  1. Mouse Tissue Dataset: Bulk RNA-seq profiling 116,512 genes across lung, kidney, liver, and small intestine tissues.
  2. Human Prostate Cancer Cell Line Dataset: Data from 16,247 genes across four cell lines, comparing samples sensitive to chemotherapy versus those that are resistant.

The data was normalized using FPKM (Fragments Per Kilobase of transcript per Million mapped reads) and log-transformed before clustering.

Key Insights and Ideas

  • Breaking the Parametric Habit: The "aha moment" here is that we don't need to force biological data into a Poisson or Negative Binomial box. By letting the data dictate the shape of the distribution, the model captures nuances that parametric models miss.
  • Robustness to "Zero-Inflation": In the prostate cancer dataset, the authors found that the algorithm struggled with the high number of zeros. They successfully implemented a simple imputation strategy (replacing zeros with tiny random values), showing the method's robustness when adapted to sparse data.
  • Fundamentally Different Solutions: The authors used the Adjusted Rand Index (ARI) to compare their results to traditional methods. The low ARI scores indicate that npMSL isn't just "slightly better"—it is finding entirely different patterns of gene co-expression that parametric methods are blind to.

Key Results

The primary metric for success was Gene Ontology (GO) enrichment—essentially checking if the genes in a cluster actually perform similar biological functions.

  • Mouse Dataset: npMSL identified 17 clusters, and 94.1% of them showed significant biological enrichment. This outperformed the logit-transformation method (88%).
  • Prostate Cancer Dataset: This is where the method truly shined. npMSL identified 17 clusters with 82.4% biological enrichment. In comparison, the standard Poisson method only managed 27.8%, and the transformation method hit 34.2%.
  • Biological Specificity: The clusters weren't just random groups; they were linked to highly specific processes like water homeostasis, T-cell activation, and mRNA catabolic processes.

Limitations and Future Work

The authors acknowledge two main areas for improvement:

  1. The Independence Assumption: Currently, the model assumes samples are independent within a cluster. In reality, biological replicates are often correlated. The authors suggest that future versions could use copula functions to model these explicit dependencies without losing the nonparametric flexibility.
  2. Bandwidth Selection: The current model uses a constant bandwidth for all clusters. While this keeps the math stable, moving to component-specific or iterative bandwidth selection could potentially lead to even sharper cluster definitions, though it risks breaking the algorithm's guaranteed convergence.

The code for this project is publicly available on GitHub.

Summary updated

Abstract

Identification of clusters of co-expressed genes in transcriptomic data is a difficult task. Most algorithms used for this purpose can be classified into two broad categories: distance-based or model-based approaches. Distance-based approaches typically utilize a distance function between pairs of data objects and group similar objects together into clusters. Model-based approaches are based on using the mixture-modeling framework. Compared to distance-based approaches, model-based approaches offer better interpretability because each cluster can be explicitly characterized in terms of the proposed model. However, these models present a particular difficulty in identifying a correct multivariate distribution that a mixture can be based upon. In this manuscript, we review some of the approaches used to select a distribution for the needed mixture model first. Then, we propose avoiding this problem altogether by using a nonparametric MSL (Maximum Smoothed Likelihood) algorithm. This algorithm was proposed earlier in statistical literature but has not been, to the best of our knowledge, applied to transcriptomics data. The salient feature of this approach is that it avoids explicit specification of distributions of individual biological samples altogether, thus making the task of a practitioner easier. When used on a real dataset, the algorithm produces a large number of biologically meaningful clusters and compares favorably to the two other mixture-based algorithms commonly used for RNA-seq data clustering. Our code is publicly available in Github at https://github.com/Matematikoi/non_parametric_clustering.
BibTeX
@inproceedings{lozano2023nonparametric,
  author = {Gabriel Lozano and Nadia A. Lanman and Michael Levine},
  title = {{Nonparametric clustering of RNA-sequencing data}},
  booktitle = {Statistical},
  year = {2023},
  doi = {10.1002/sam.11638},
}
Figure