Neglecting the impact of normalization in semi-synthetic RNA-seq data simulations generates artificial false positives

Boris P. Hejblum, Kalidou Ba, Rodolphe Thiébaut, Denis AgnielGenome 2024 · DOI · OpenAlex

Introduction

A 2022 study by Li et al. sent shockwaves through the bioinformatics community by claiming that standard RNA-seq differential expression (DE) tools, such as edgeR and DESeq2, fail to control the False Discovery Rate (FDR) as sample sizes grow into the hundreds. They recommended the non-parametric Wilcoxon rank-sum test as a more robust alternative for large-scale population studies. However, a follow-up study published in Genome Biology by Hejblum et al. identifies a critical flaw in the original simulation design that led to these alarming conclusions.

The Normalization Artifact

The core of the issue lies in the order of operations during data simulation. In RNA-seq, normalization is required to account for differences in "library size" (sequencing depth). The original study generated "null" data—where no biological difference should exist—by permuting raw read counts between samples.

Hejblum et al. argue that permuting raw counts before normalization creates a "library size trap." If a gene's raw count from a sample with a large library is swapped with one from a sample with a small library, the subsequent normalization process creates an artificial outlier. The DE tools were not failing; they were correctly identifying these mathematically induced artifacts as significant differences.

Figure B: Comparison of FDR control across different simulation schemes. The "Permutations first" approach (top row) leads to inflated false positives, whereas the "Normalization first" approach (bottom row) restores correct FDR control across various sample sizes.

Correcting the Simulation

To test this hypothesis, the authors reproduced the original simulation using GTEx project data (Heart atrial appendage and left ventricle samples) and introduced an "amended" scheme. In this corrected approach, the data is normalized first, and then the values are permuted to create a true null hypothesis that respects the relative expression levels.

They benchmarked several methods, including dearseq (asymptotic and permutation-based), limma-voom, edgeR, DESeq2, NOISeq, and the Wilcoxon rank-sum test, across three scenarios: the original flawed permutation, a "no normalization" scenario, and the corrected "normalization-first" permutation.

Key Results

Under the amended simulation scheme, the "alarming" false positives vanished. Most popular tools, including dearseq, limma-voom, and NOISeq, successfully controlled the FDR even with hundreds of samples.

Figure A: Empirical vs. Nominal FDR for various differential expression methods using the corrected simulation scheme at a large sample size (n=372/386 total). Most methods maintain good error control, following the diagonal.

Furthermore, the study challenges the supposed superiority of the Wilcoxon test. When applied to the same normalized data, the Wilcoxon test also "failed" under the flawed simulation. Under the correct simulation, dearseq (asymptotic) demonstrated higher statistical power than the Wilcoxon test once sample sizes exceeded approximately 20 per group.

Conclusion

This research serves as a vital "correction of the record," reassuring researchers that standard RNA-seq tools remain reliable for large-scale studies. It highlights the critical importance of simulation design: neglecting the mathematical implications of normalization can lead to "garbage in, garbage out" results that misguide the community. Tools like dearseq not only maintain error control but also offer greater flexibility for complex experimental designs compared to simple non-parametric tests.

Summary updated

Abstract

A recent study reported exaggerated false positives by popular differential expression methods when analyzing large population samples. We reproduce the differential expression analysis simulation results and identify a caveat in the data generation process. Data not truly generated under the null hypothesis led to incorrect comparisons of benchmark methods. We provide corrected simulation results that demonstrate the good performance of dearseq and argue against the superiority of the Wilcoxon rank-sum test as suggested in the previous study.
BibTeX
@inproceedings{hejblum2024neglecting,
  author = {Boris P. Hejblum and Kalidou Ba and Rodolphe Thiébaut and Denis Agniel},
  title = {{Neglecting the impact of normalization in semi-synthetic RNA-seq data simulations generates artificial false positives}},
  booktitle = {Genome},
  year = {2024},
  doi = {10.1186/s13059-024-03231-9},
}
Figure