A nonparametric approach for relevance determination

Babak Shahbaba

Abstract

The objective of many high-throughput studies is to identify factors that are relevant to an outcome of interest. Such studies are abundant in genetics, image processing, astrophysics, and neuroscience. In this paper, we argue that treating these problems as large-scale hypothesis tests does not reflect the motivation behind the studies. Instead, we suggest treating these studies as a decision problem, where our primary concern is selecting the most relevant set of factors for a more focused follow-up. Furthermore, instead of dividing the factors into a significant and a non-significant group, we propose a flexible Bayesian model that accommodates subgroups with different degrees of importance. To this end, our approach uses a simple extension of Dirichlet process mixtures to model the relationship between the factors and the outcome. With simulated data, we demonstrate that our model performs substantially better than alternative methods based on the false discovery rate. We also apply our method to two real large-scale studies: one interrogating the mutation status of p53 in cancer cell lines, and another identifying differentially expressed genes in leukemia. Overall, we find that accounting for the true motivation behind high-throughput studies and the complexity of relationships between factors and outcomes improves analysis by increasing the discovery rate of relevant factors while maintaining a low false discovery rate.

Introduction

In modern biological research, high-throughput studies—such as microarrays—measure thousands of variables simultaneously across a relatively small number of samples. The standard statistical paradigm for analyzing this data involves performing thousands of individual hypothesis tests, followed by a correction method like the False Discovery Rate (FDR) to mitigate the multiple testing problem.

However, this traditional approach often fails to align with the actual goals of the researcher. In practice, these studies are exploratory; the objective is not a definitive binary classification of every gene, but rather the identification of a manageable subset of "relevant" factors for expensive, labor-intensive follow-up experiments. Furthermore, traditional methods typically ignore the shared information across genes and struggle to identify factors that may be relevant due to changes in variability rather than just mean expression levels.

Methodology: Bayesian Relevance Determination (BRD)

To address these limitations, we propose Bayesian Relevance Determination (BRD), a hierarchical Bayesian model that treats variable selection as a clustering problem rather than a series of independent tests.

The Core Model

For each factor (e.g., a gene), we model the expression level using a linear regression framework where the coefficient $\beta_i$ represents the effect size of the condition or disease. Instead of assuming all $\beta_i$ are drawn from a single distribution, we employ Adaptive Shrinkage. We assume each $\beta_i$ is drawn from a normal distribution with a specific variance $\tau_i^2$.

Dirichlet Process Mixtures (DPM)

The defining feature of our approach is the use of a Dirichlet Process prior on the variances ($\tau_i^2$). Because the Dirichlet Process is discrete, it naturally clusters factors into a small number of subgroups based on their degree of relevance. This nonparametric approach allows the data to determine the number of relevance tiers—such as "noise," "moderately relevant," and "highly relevant"—without requiring the researcher to specify the number of groups in advance.

Ranking and Prioritization

Factors associated with very small variances are clustered as noise, while those in clusters with larger variances are flagged as relevant. By clustering on the variance of the effect size while fixing the mean at zero, the model simultaneously detects genes with large shifts in mean expression and those that become more volatile under disease conditions.

Experimental Results

Simulated Data

Using simulations based on p53 cancer cell line data, we compared the BRD model against alternative methods, including locFDR and the Bayesian Discovery Procedure (BDP). The BRD model consistently achieved a higher Area Under the Curve (AUC) in ROC analysis, demonstrating a superior ability to surface true positives while maintaining a low false discovery rate.

Leukemia Benchmark

We applied the model to a benchmark dataset containing expression levels of 10,056 genes used to differentiate between Acute Myeloid Leukemia (AML) and Acute Lymphoid Leukemia (ALL). The BRD model identified a highly parsimonious set of just five genes as "highly relevant": TCL1A, DNTT, CD24, TOP2B, and PSMA6. Notably, all five genes have well-documented links to leukemia in existing literature, validating the model's ability to find the "needle in the haystack" within massive datasets.

HCMV Infection

In a study of 12,626 genes comparing cells infected with human cytomegalovirus (HCMV) against a control group, the model successfully identified relevant factors while borrowing strength across the entire gene set, leading to more stable estimates than isolated t-tests.

Discussion

The primary advantage of BRD is its move away from binary significance. By acknowledging that relevance exists on a spectrum, the model provides a more nuanced ranking for researchers. Additionally, the Bayesian framework allows for "information borrowing," where the collective behavior of thousands of genes informs the estimation of individual parameters.

Limitations and Future Directions

The main drawback of the current implementation is computational complexity. The model relies on Markov Chain Monte Carlo (MCMC) sampling, which is significantly slower than frequentist t-tests. For the leukemia dataset, the process required approximately 3 seconds per iteration. Future work will focus on improving efficiency through split-merge MCMC algorithms or Maximum A Posteriori (MAP) estimates. Furthermore, we aim to extend the model to incorporate prior biological knowledge, such as gene pathways, to further refine the relevance determination process.

Conclusion

By reframing high-throughput analysis as a decision problem rather than a hypothesis testing problem, Bayesian Relevance Determination offers a more effective way to prioritize factors for follow-up research. Our results suggest that accommodating the inherent complexity of biological relationships leads to higher discovery rates and more biologically meaningful insights.

Summary updated

Abstract

The objective of many high throughput studies is to identify factors that are relevant to an outcome of interest. Such studies are abundant in genetics, image processing, astrophysics, and neuroscience. In this paper, we argue that treating these problems as large-scale hypothesis tests does not reflect the motivation behind the studies. Instead, we suggest treating these studies as a decision problem, where our primary concern is selecting the most relevant set of factors for a more focused follow up. Furthermore, instead of dividing the factors into a significant and a non-significant group (which is common in the hypothesis testing framework), we propose a flexible Bayesian model that accommodates subgroups with different degrees of importance. To this end, our approach uses a simple extension of Dirichlet process mixtures to model the relationship between the factors and the outcome. With simulated data, we demonstrate that our model performs substantially better than alternative methods based on the false discovery rate. We also apply our method to two real large-scale studies. The objective of the first study is to interrogate the mutation status of p53 in cancer cell lines, and the objective of the second study is to identify differentially expressed genes between two types of leukemia. Overall, we find that taking into account the true motivation behind high throughput studies and the possible complexity of relationships between factors and outcome could improve the analysis by increasing the rate of discovering relevant factors while keeping the false discovery rate low.
BibTeX
@misc{shahbaba2010nonparametric,
  author = {Babak Shahbaba},
  title = {{A nonparametric approach for relevance determination}},
  year = {2010},
  eprint = {1003.2390},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1003.2390},
}
Figure