Bayesian genome-wide clustering and variable selection of transcriptomic data via rank-based mixtures

Emilie Eliseussen, Haakon Muggerud, Luca Coraggio, Ida Scheel, Thomas Fleischer, Valeria VitelliarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces lowBM3, a novel Bayesian framework for clustering and identifying relevant genes in high-dimensional transcriptomic data using rankings. It addresses the scalability limitations of existing methods, enabling more robust analysis of complex biological datasets like those from cancer genomics.

What Problem Does This Paper Address?

Analyzing high-dimensional biological data, such as transcriptomics (gene expression levels), often involves dealing with complex patterns and identifying subsets of genes that are crucial for understanding biological processes or disease subtypes. Traditional methods struggle with the sheer number of genes and the inherent noise or batch effects in such data. While rank-based methods offer robustness by transforming continuous measurements into ordered lists, existing Bayesian approaches like the Bayesian Mallows Model (BMM) don't scale well to the ultra-high dimensions found in genomics. Furthermore, these methods often assume a single underlying structure (e.g., one consensus ranking) and don't effectively handle the heterogeneity often present in patient populations (e.g., different cancer subtypes). This paper tackles the challenge of developing a scalable, Bayesian framework that can simultaneously cluster samples, select relevant genes, and account for this heterogeneity in high-dimensional ranking data.

Methods

The core contribution is the lowBM3 (lower-dimensional Bayesian Mallows Model Mixture) model. It builds upon the lowBMM (lower-dimensional Bayesian Mallows Model) by extending it to a finite mixture model. lowBMM itself addresses scalability by focusing on a lower-dimensional subset of "relevant" items (genes) and assuming the rest are noise. lowBM3 introduces C mixture components, each with its own set of relevant genes (A*) and a cluster-specific consensus ranking (p_c). This allows for modeling heterogeneity within the data.

Key methodological aspects include:

  • Rank-based approach: Transforms continuous gene expression data into rankings, making it robust to non-normal distributions and batch effects.
  • Bayesian inference: Provides uncertainty quantification for all estimates, crucial for biological interpretation.
  • Variable selection: Explicitly identifies a subset of "relevant" genes for each cluster, reducing dimensionality and improving interpretability.
  • Mixture modeling: Accommodates multiple underlying biological subtypes or clusters within the data.
  • MCMC algorithm: A sophisticated Markov Chain Monte Carlo scheme is developed to efficiently sample from the posterior distribution of model parameters.
  • Postprocessing framework: Novel methods are introduced to summarize posterior distributions of discrete parameters like cluster assignments and relevant gene sets, providing interpretable results.

Datasets

The paper evaluates lowBM3 using simulated datasets generated from various data-generating processes (DGPs) designed to mimic different scenarios of ranking data, including "top-rank" (relevant items are ranked highest) and "consistency" (relevant items have contiguous ranks) designs, with and without added noise.

The primary real-world dataset used is RNA-seq data from breast cancer patients obtained from the TCGA Data Portal. This dataset is characterized by ultra-high dimensionality (15,348 genes) and a significant number of patients (845).

Key Insights and Ideas

  • Scalability for Genomics: The primary breakthrough is enabling Bayesian clustering and variable selection for ultra-high-dimensional transcriptomic data, a significant hurdle for previous methods.
  • Heterogeneity is Key: Explicitly modeling multiple clusters (lowBM3) is crucial for accurately representing biological data, which often contains distinct patient subtypes (e.g., breast cancer subtypes).
  • Rankings as a Robust Feature: Using rankings, rather than raw expression values, effectively handles noise and batch effects, leading to more stable and interpretable results.
  • Joint Clustering and Variable Selection: The integrated approach of lowBM3 provides a more parsimonious and accurate model by simultaneously identifying clusters and the genes that define them.
  • Biomarker Discovery Potential: The method's ability to identify cluster-specific relevant genes offers a powerful tool for discovering biomarkers that characterize different biological states or disease subtypes.

Key Results

  • Simulation Studies: lowBM3 demonstrated high accuracy in clustering assignments and identifying relevant genes across various simulated scenarios, including those with noise and different data-generating processes. It showed robustness to misspecification of certain parameters (like the scale parameter α).
  • Breast Cancer Data Application: The analysis of BRCA RNA-seq data revealed biologically meaningful clusters that showed correspondence with known PAM50 breast cancer subtypes. The method successfully identified sets of genes enriched in specific pathways (e.g., Estrogen Response) for different clusters, providing insights into the molecular mechanisms underlying these subtypes. For instance, the gene GATA3 was identified as a key driver in several clusters associated with luminal breast cancer.

Limitations and Future Work

  • Complete Data Assumption: The current implementation assumes complete rankings. The authors suggest that data augmentation techniques, as used in prior work, could be incorporated to handle missing data.
  • Fixed Parameters: The scale parameter α and the dimensionality n* are fixed. Estimating these within the inferential procedure would add complexity but could improve flexibility.
  • Tuning Parameters: While the MCMC algorithm is efficient, further exploration of adaptive methods for tuning parameters like L and Le could be beneficial.
  • Number of Clusters (C): lowBM3 requires specifying C in advance. While an elbow criterion was proposed and evaluated, future work could explore infinite mixture models or Bayesian nonparametric approaches for automatic estimation of C.
  • Computational Cost: While more scalable than BMM, the method still involves computationally intensive MCMC sampling, especially for very large datasets.

The authors suggest that lowBM3 is a significant step towards unsupervised analysis of ultra-high-dimensional ranking data and has strong potential for biomarker discovery in complex biological datasets.

Summary updated

Abstract

With the increasing availability of ranking data, there has been a growing demand for appropriate unsupervised rank-based inferential frameworks capable of handling high-dimensional datasets and providing uncertainty quantification for all estimates. Rank-based methods have also seen a growing popularity in -omics pipelines, as ranking continuous measurements provides a robust means of handling non-normally distributed data. The Bayesian Mallows model (BMM) has emerged as a promising choice because of its adaptability to various types of ranking data and its flexible framework, integrating cluster-wise rank aggregation with inference at the individual level. However, the scalability of BMM to ultra-high-dimensional settings, such as -omics analyses, has remained limited. The present paper addresses this issue by introducing the first rank-based model generalizing BMM to jointly handle clustering and variable selection, namely the lower-dimensional Bayesian Mallows Model Mixture (lowBM3). The proposed method provides a novel Bayesian framework that simultaneously handles heterogeneity in the sample, unsupervised parameter estimation, and model selection in a scalable manner for ultra-high-dimensional data. Additionally, a companion postprocessing framework is introduced to provide posterior summaries of the discrete posterior distributions of both the consensus ranking and the variable selector. Simulation studies are performed to assess the performance of the method. The usefulness of the method is also shown in an application to signature discovery for cancer genomics, where RNA-seq bulk gene expression data obtained from breast cancer patients are clustered genome-wide.
BibTeX
@misc{eliseussen2026bayesian,
  author = {Emilie Eliseussen and Haakon Muggerud and Luca Coraggio and Ida Scheel and Thomas Fleischer and Valeria Vitelli},
  title = {{Bayesian genome-wide clustering and variable selection of transcriptomic data via rank-based mixtures}},
  year = {2026},
  eprint = {2606.06730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.06730},
}

Loading related content...

Figure