TL;DR
This paper tackles the challenge of choosing the right analytical workflow for data-independent acquisition (DIA) proteomics, especially for clinical samples with inherent patient variability. By creating a large, heterogeneous benchmark dataset and testing over 1400 workflow combinations, they reveal that spectral library generation and DIA software choice have the biggest impact on results, while non-parametric statistical tests consistently perform best.
What Problem Does This Paper Address?
The field of proteomics, particularly using mass spectrometry, is rapidly advancing. Data-independent acquisition (DIA) is a powerful technique for quantifying proteins across many samples, but it relies heavily on complex analysis pipelines. These pipelines involve numerous choices: how to generate spectral libraries (which act as a reference for identifying proteins), which software to use for data analysis, how to handle missing data points (sparsity reduction), how to normalize intensity values, and which statistical tests to apply to find differentially abundant proteins. With so many options, researchers, especially those working with clinical samples exhibiting significant inter-patient variability, struggle to select the optimal workflow. Existing benchmarks often use simplified datasets that don't reflect real-world complexity, making it hard to guide these critical decisions. This paper aims to provide a comprehensive, robust benchmark using a realistic dataset to help researchers navigate this complex analytical landscape.
Methods
The authors developed a sophisticated benchmarking framework. They created a large dataset by spiking E. coli peptides (a known ground truth) into human lymph node samples from 92 individuals, capturing real inter-patient heterogeneity. This dataset was then used to evaluate a vast number of data analysis workflows.
The core of their methodology involved:
- DIA Data Acquisition: Standard DIA acquisition was performed on the prepared samples.
- Spectral Library Generation: They explored three main strategies:
- Using spectral libraries generated from pre-fractionated DDA runs.
- Refining in-silico predicted spectral libraries (using tools like DIA-NN and PROSIT) with gas-phase fractionation (GPF).
- Using purely in-silico predicted libraries.
- DIA Software Analysis: Four popular DIA analysis software suites were tested: DIA-NN, Skyline, OpenSwath, and Spectronaut.
- Data Preprocessing: They investigated combinations of:
- Sparsity Reduction: No reduction, requiring >66% or >90% of values per protein.
- Normalization: Unnormalized, quantile normalization (QN), tail-robust quantile normalization (TRQN), and median normalization.
- Statistical Testing: Seven different statistical tests were applied to identify differentially abundant proteins.
- Bootstrapping: To ensure robustness and assess the impact of sample size, they generated 2100 bootstrap datasets by resampling from the original data, varying group sizes from 3 to 23.
This systematic combination resulted in 1428 distinct data analysis workflows that were evaluated.
Datasets
The primary dataset used in this study was a newly created benchmark dataset. It consisted of:
- Human Samples: Formalin-fixed paraffin-embedded (FFPE) lymph node tissue from 92 individuals, representing real inter-patient heterogeneity.
- Spike-in: Escherichia coli (E. coli) peptides were added to the human samples at four different ratios: "human only" (control), 1:25, 1:12, and 1:6 (E. coli:human). This provided a known ground truth for evaluating performance.
The study also utilized various spectral libraries, including those generated from DDA runs and in-silico predicted libraries (DIA-NN and PROSIT), some of which were refined using gas-phase fractionation (GPF).
Key Insights and Ideas
- The "Workflow" is King: The choice of DIA analysis software and the spectral library it uses has the most significant impact on the accuracy of identifying differentially abundant proteins. This highlights that the foundational elements of the analysis pipeline are paramount.
- Spectral Libraries Matter, Especially GPF-Refined: Using gas-phase fractionated (GPF) spectral libraries generally improved performance, particularly when compared to purely in-silico predicted libraries. This suggests that experimental refinement of libraries, even for predicted ones, is beneficial.
- Non-Parametric Tests Reign Supreme: Among the statistical tests, non-parametric, permutation-based methods like SAM and ROTS consistently outperformed others, especially for robustly identifying true differences.
- Normalization is Less Critical (in this context): Surprisingly, most normalization methods did not improve performance and sometimes even decreased it. This might be due to the specific nature of the benchmark dataset (unbalanced differential expression) and the assumptions of the normalization methods.
- Sparsity Reduction Trade-offs: While removing missing values (sparsity reduction) can clean up data, overly aggressive filtering led to a decrease in performance, likely by removing true positives.
- Missing Value Handling is Complex: Different software suites handled missing values differently, with some correctly identifying true absences (e.g., E. coli in human-only samples) and others requiring specific tools (like TRIC for OpenSwath) to achieve this.
Key Results
- 1428 Workflows Evaluated: The study systematically tested a vast number of combinations of spectral libraries, DIA software, sparsity reduction, normalization, and statistical tests.
- DIA Software & Library Impact: The choice of DIA software and spectral library significantly influenced the number of identified proteins and the accuracy of quantification. Spectronaut's "DirectDIA" performed well when evaluated against its own workflow's protein list, while DIA-NN combined with a GPF-refined library excelled when considering a broader set of identified proteins.
- Statistical Test Performance: Non-parametric permutation-based tests (SAM, ROTS) consistently showed the best prediction performance (measured by partial Area Under the Curve - pAUC).
- Sparsity Reduction Impact: No sparsity reduction generally yielded the highest pAUC, indicating that aggressive filtering can reduce sensitivity.
- Normalization Impact: Normalization methods generally showed a minor impact, often leading to a slight decrease in performance.
Limitations and Future Work
- Dataset Specificity: While the dataset captures inter-patient heterogeneity, it's still a benchmark. The authors acknowledge that other proteomes (e.g., plasma) might require dedicated benchmarking.
- Software Versions: The study used specific versions of software, and newer versions might yield different results. For instance, the "match-between-runs" function was not yet included in the DIA-NN version used.
- Normalization Assumptions: The observed performance of normalization methods might be an artifact of the study design, which featured unbalanced differential expression. Future work could explore normalization strategies better suited for such scenarios.
- Future Directions: The authors aim to further investigate the connection between data characteristics (variance, skewness, etc.) and algorithm performance to provide more tailored algorithm recommendations for researchers. They also encourage others to use their dataset to assess their own workflows, especially in clinical settings.
Summary updated