The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies

Sheng Zhong, Leming M. Shi, Wendell D. Jones, Roderick V. Jensen, Stephen C. Harris, Roger Perkins, Federico M. Goodsaid, Lei Guo, Lisa J. Croner, Cecilie Boysen, Hong Fang, Feng Qian, Shashi Amur, Wenjun Bao, Catalin C. Barbacioru, Vincent Bertholet, Xiaoxi Megan Cao, Tzu-Ming Chu, Patrick J. Collins, Xiao-hui Fan, Felix W. FruehBMC Bioinformatics 2008 · DOI

TL;DR

Microarray studies often struggle with reproducible lists of differentially expressed genes (DEGs). This paper demonstrates that relying solely on statistical significance (P-value) leads to poor reproducibility, especially with small sample sizes. Instead, ranking genes by fold-change (FC) in combination with a non-stringent P-value cutoff significantly improves DEG list reproducibility across different platforms and labs, offering a practical guideline for more reliable gene expression analysis.

What Problem Does This Paper Address?

The reliability of microarray experiments has been questioned due to the lack of reproducibility in identifying differentially expressed genes (DEGs) across similar studies. This inconsistency leads to confusion and debate within the scientific community regarding the appropriate methods for DEG identification. While new statistical methods are constantly emerging, the fundamental challenge lies in generating DEG lists that are not only sensitive and specific but also consistently reproducible. This paper aims to address this by investigating the impact of different gene selection strategies on DEG list reproducibility, using data from the MicroArray Quality Control (MAQC) project.

Methods

The authors employed a comprehensive approach to evaluate gene selection methods:

  • Inter-site Comparisons: Analyzing data from the same microarray platform across different laboratories to assess reproducibility within a platform.
  • Cross-platform Comparisons: Comparing results from different microarray platforms using the same RNA samples to evaluate platform-specific variability.
  • Microarray vs. TaqMan® Assays: Validating microarray findings against TaqMan® real-time PCR, a widely accepted "gold standard" for gene expression measurement.
  • Gene Selection Rules: Investigating several methods for identifying DEGs, primarily focusing on:
  • P-ranking: Ranking genes based on statistical significance (P-value) from a simple two-tailed t-test.
  • FC-ranking: Ranking genes based on the magnitude of fold-change (FC) in expression levels.
  • Combined FC and P-ranking: Using both FC and a non-stringent P-value cutoff.
  • Reproducibility Metric: The primary metric used was the Percentage of Overlapping Genes (POG), which quantifies the similarity between DEG lists generated by different methods or sites.
  • Noise Filtering: Analyzing data both with and without filtering out "absent" calls to assess the impact of noise on reproducibility.
  • Simulated Datasets: Creating simulated datasets with varying levels of biological variability and noise to parametrically evaluate gene selection methods under controlled conditions.

Datasets

The primary datasets used in this study were generated by the MicroArray Quality Control (MAQC) project. This project involved:

  • Two distinct RNA samples: Stratagene Universal Human Reference RNA (MAQC sample A) and Ambion Human Brain Reference RNA (MAQC sample B).
  • Five commercial whole-genome microarray platforms: ABI, AFX, AG1, GEH, and ILM.
  • Technical replicates: Five replicates per sample per platform per laboratory.
  • TaqMan® assays: Used to validate a subset of 906 genes.

The study also utilized simulated datasets (MAQC-simulated, Small-Delta, and Medium-Delta) to control for biological variability and noise, allowing for a more direct assessment of gene selection method performance.

Key Insights and Ideas

  • P-value alone is insufficient for reproducibility: Relying solely on statistical significance (P-value) from simple t-tests leads to highly variable and irreproducible DEG lists, especially when selecting a small number of genes. This is a mathematical consequence of the inherent variability of t-values with small sample sizes.
  • Fold-change is a robust ranking criterion: Ranking genes by fold-change (FC) consistently yields much higher reproducibility (POG) compared to P-ranking, even with small gene lists. FC directly measures the biological effect size.
  • The "sweet spot" for reproducibility: Combining FC-ranking with a non-stringent P-value cutoff offers a balanced approach, enhancing reproducibility while maintaining reasonable sensitivity and specificity.
  • Noise filtering has a modest impact: While noise filtering can slightly improve POG and stability, it does not fundamentally alter the observed trends or the superiority of FC-ranking for reproducibility.
  • Perception of reproducibility is method-dependent: The apparent lack of reproducibility in many published microarray studies is largely an artifact of the chosen gene selection methodology, not necessarily an inherent flaw of the technology itself.

Key Results

  • FC-ranking vs. P-ranking: Inter-site and cross-platform POG for FC-ranking reached near 90% with as few as 20 genes, while P-ranking struggled to reach this level even with thousands of genes.
  • Microarray vs. TaqMan®: Concordance between microarray and TaqMan® assays was significantly higher when using FC-ranking (80-85% with noise filtering) compared to P-ranking.
  • Simulated data: FC-ranking, especially when combined with a P-value cutoff, consistently outperformed P-ranking alone in terms of reproducibility across various simulated scenarios, particularly when biological variability (CV) increased.
  • Volcano plots: The study visually demonstrated the utility of volcano plots for selecting DEGs by jointly considering FC and P-value, recommending FC as the primary ranking criterion.

Limitations and Future Work

  • "Truth" is elusive: The study acknowledges that in real-world experiments, the "truth" (i.e., the actual set of differentially expressed genes) is often unknown. Therefore, reproducibility, cross-platform concordance, and agreement with gold-standard methods like TaqMan® assays serve as proxies for reliability.
  • Stringent requirements: While the proposed FC + non-stringent P cutoff method provides a practical baseline, the authors note that robust methods are still needed for stringent clinical requirements.
  • Future directions: The paper implicitly suggests further exploration into optimal FC and P-value cutoffs for different experimental contexts and the development of more advanced statistical methods that can better balance reproducibility, sensitivity, and specificity. The authors also highlight the need for continued awareness and education on appropriate gene selection practices.

Summary updated

BibTeX
@article{zhong2008balance,
  author = {Sheng Zhong and Leming M. Shi and Wendell D. Jones and Roderick V. Jensen and Stephen C. Harris and Roger Perkins and Federico M. Goodsaid and Lei Guo and Lisa J. Croner and Cecilie Boysen and Hong Fang and Feng Qian and Shashi Amur and Wenjun Bao and Catalin C. Barbacioru and Vincent Bertholet and Xiaoxi Megan Cao and Tzu-Ming Chu and Patrick J. Collins and Xiao-hui Fan and Felix W. Frueh},
  title = {{The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies}},
  journal = {BMC Bioinformatics},
  year = {2008},
  doi = {10.1186/1471-2105-9-s9-s10},
}
Figure