SABRE: a method for assessing the stability of gene modules in complex tissues and subject populations

Casey P. Shannon, Virginia Chen, Mandeep Takhar, Zsuzsanna Hollander, Robert Balshaw, Bruce McManus, Scott J. Tebbutt, Don D. Sin, Raymond T. NgBMC Bioinformatics 2016 · DOI

TL;DR

This paper introduces SABRE, a novel method for assessing the stability of gene modules identified from transcriptomic data. SABRE uses bootstrap resampling to quantify how consistently gene modules are identified across different subsets of samples, providing a crucial metric for the reproducibility of systems biology findings, especially in heterogeneous human tissues.

What Problem Does This Paper Address?

Identifying "gene modules" – sets of genes that are coordinately expressed – is a cornerstone of systems biology. These modules are thought to represent functional biological units and are increasingly used in translational research. However, a major challenge is understanding how reliable these identified modules are. Gene expression data, especially from human tissues, is inherently noisy and subject to significant biological variation between individuals. This means that modules identified from one set of samples might not be consistently found in another, a problem known as module instability. Existing methods for assessing module reproducibility often rely on external validation datasets or network topology, which aren't always applicable. The authors aim to address this by developing a robust, internal method to quantify module stability that is independent of network structure and external validation, and to understand how this stability is affected by sample size and the choice of module discovery algorithm.

Methods

The core of this paper is the SABRE (Similarity Across Bootstrap RE-sampling) procedure. Here's how it works:

  1. Reference Module Set: First, gene modules are identified from the entire available dataset using one or more gene network inference (GNI) algorithms. This forms the "reference module set."
  2. Bootstrap Resampling: The original dataset is repeatedly resampled with replacement to create many smaller "bootstrap" datasets.
  3. Module Discovery on Resamples: Each GNI algorithm is then used to identify modules within each of these bootstrap datasets.
  4. Similarity Assessment: For each module in the reference set, its similarity to modules found in the bootstrap sets is calculated. The authors propose a modified Jaccard similarity coefficient, termed Similarity(q, q') = |q ∩ q'| / min(|q|, |q'|), which accounts for cases where a module is perfectly contained within another.
  5. Stability Metric (H-index): To summarize the stability of a reference module, the distribution of its best-match similarity scores across all bootstrap resamples is used. This distribution is then summarized by the Hirsch-index (H-index), defined as the maximum value 'h' such that the module's similarity score was greater than or equal to 'h' in at least 'h' proportion of bootstrap runs. A module with an H-index of 0.8 means it achieved at least 0.8 similarity to its best match in 80% of the bootstrap samples.

The authors also explored:

  • Random Gene Modules: To establish a baseline, they created random modules from gene lists and assessed their stability using SABRE.
  • Network Topology Measures: They investigated the correlation between module stability and various network properties like the average number of neighbors and shortest path appearances.
  • Functional Annotation: They assessed whether stable modules were more likely to be enriched in known biological pathways and gene sets (MSigDB and BTMs).

Datasets

The primary dataset used in this study consists of gene expression profiles from 238 peripheral whole blood samples of patients with chronic obstructive pulmonary disease (COPD), collected as part of the Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) study. This dataset is noted for being from a highly heterogeneous tissue and a complex human disease.

Key Insights and Ideas

  • Stability is Crucial for Reproducibility: The core idea is that stable modules are more likely to be reproducible across different sample subsets, making them more reliable for biological interpretation and further study.
  • SABRE Provides a Quantifiable Metric: The H-index offers a single, interpretable number to rank modules by stability, moving beyond qualitative assessments.
  • Sample Size Matters: Module stability generally increases with sample size, suggesting that larger cohorts are essential for robust module identification in complex biological systems.
  • Algorithm Choice Impacts Stability: Different GNI algorithms yield modules with varying degrees of stability, highlighting the importance of considering the method used.
  • Stable Modules Map to Biology: Stable modules are more likely to correspond to known biological functions and pathways, reinforcing their biological relevance.

Key Results

  • Increased Sample Size Enhances Stability: Module stability, as measured by the H-index, consistently increased with larger sample sizes, with a plateau appearing around 80-120 samples.
  • WGCNA Modules More Stable: Modules identified by the Weighted Gene Co-expression Network Analysis (WGCNA) algorithm generally showed higher stability (mean H-index = 0.79) compared to those from the Pihur (mean H-index = 0.68) or Chaussabel (mean H-index = 0.69) approaches.
  • Random Modules are Unstable: Randomly assembled modules had very low stability (H-index ≈ 0.25), serving as a useful baseline. Even for small studies (n=10), WGCNA modules showed better worst-case stability than random modules.
  • Stability and Connectivity: A positive association was observed between module stability and certain network connectivity measures (e.g., number of shortest paths), though this relationship was strongest in modules identified by the Chaussabel approach, which did not explicitly consider topology.
  • Annotatability: Stable modules were significantly more likely to be enriched in known biological pathways and gene sets across various collections (e.g., MSigDB Hallmark, BTMs).
  • Novel Module Discovery: The study identified two highly stable, yet initially un-annotated, modules (lightgreen and blue) that were later found to correspond to known biological processes relevant to COPD, validating both the modules and the SABRE ranking.

Limitations and Future Work

  • Sample Size Threshold: While stability increases with sample size, the paper notes that a plateau is reached around 80-120 samples, suggesting that beyond this point, further increases may yield diminishing returns for stability.
  • Algorithm Comparison Nuances: The paper acknowledges that the observed differences in stability between algorithms might be influenced by the fact that WGCNA and the Pihur approach use average linkage hierarchical clustering on an adjacency matrix, while the Chaussabel approach uses k-means clustering.
  • Future Work: The authors suggest that SABRE and the H-index can guide the selection of modules for further experimental investigation. They also propose that comparing module stability across different conditions could reveal loss of regulation, which could be of biological interest. The findings also suggest that applying multiple GNI strategies to a dataset and comparing their stable modules could be a valuable approach.

Summary updated

BibTeX
@article{shannon2016sabre,
  author = {Casey P. Shannon and Virginia Chen and Mandeep Takhar and Zsuzsanna Hollander and Robert Balshaw and Bruce McManus and Scott J. Tebbutt and Don D. Sin and Raymond T. Ng},
  title = {{SABRE: a method for assessing the stability of gene modules in complex tissues and subject populations}},
  journal = {BMC Bioinformatics},
  year = {2016},
  doi = {10.1186/s12859-016-1319-8},
}
Figure