Abstract
Synthetic lethality (SL) is a type of genetic interaction that occurs when defects in two genes cause cell death, while a defect in a single gene does not. Targeting an SL partner of a gene mutated in cancer can selectively kill tumor cells. Traditional wet-lab experiments for SL screening are resource-intensive. Hence, many computational methods have been developed for virtual screening of SL gene pairs. This study benchmarks recent machine learning methods for SL prediction, including three matrix factorization and eight deep learning models. We scrutinize model performance using various data splitting scenarios, negative sample ratios, and negative sampling methods on both classification and ranking tasks to assess the models’ generalizability and robustness. Our benchmark analyzed performance differences among the models and emphasized the importance of data and real-world scenarios. Finally, we suggest future directions to improve machine learning methods for SL discovery in terms of predictive power and interpretability.
Introduction
In the quest for precision oncology, synthetic lethality (SL) has emerged as a promising strategy. By identifying gene pairs where the loss of both leads to cell death but the loss of either alone is tolerated, researchers can develop therapies that selectively target cancer cells harboring specific mutations. However, the sheer number of possible gene combinations—approximately 200 million in the human genome—makes exhaustive experimental screening impractical.
While numerous machine learning (ML) models have been proposed to predict SL interactions, they are often evaluated under idealized conditions that fail to reflect clinical or experimental reality. This study provides the first comprehensive benchmark of 11 state-of-the-art ML models, evaluating their performance across realistic, high-stakes scenarios.
The Benchmarking Framework
The researchers evaluated 11 models categorized into two main groups: Matrix Factorization (MF) and Deep Learning (DL), specifically Graph Neural Networks (GNNs). The benchmark was designed to test these models' generalizability, robustness, and sensitivity to data preparation strategies.
The evaluation framework focused on three critical dimensions:
- Generalizability (Data Splitting Methods - DSMs): Models were tested on three levels of difficulty: CV1 (both genes seen in training), CV2 (one gene unseen), and CV3 (neither gene seen in training, a "total cold start").
- Robustness (Positive-to-Negative Ratios - PNRs): The ratio of known SL pairs to unknown pairs was varied from 1:1 to 1:50 to simulate the "needle in a haystack" nature of SL discovery.
- Negative Sampling (NSMs): Since confirmed non-SL pairs are scarce, the study compared random selection against strategies informed by gene expression and dependency scores.
Data Integration
To power these models, the study integrated diverse biological datasets, including SynLethDB 2.0 for known SL interactions, BioGRID for protein-protein interaction networks, and functional annotations from Gene Ontology (GO) and KEGG. Additionally, data from the Cancer Dependency Map (DepMap) was used to refine negative sampling strategies through gene expression and CRISPR knockout dependency scores.
Performance Analysis and the "Cold Start" Challenge
A primary finding of the benchmark is the significant performance drop in "cold start" (CV3) scenarios. Most models appear to rely on the local network neighborhood of known genes rather than learning generalized biological principles of synthetic lethality.
As shown in the score distributions, while models like SLMGAE maintain some separation between positive and negative samples even in CV3, others like GCATSL see their predictive power collapse, with scores for all sample types clustering together in a narrow spike. Despite this, SLMGAE (a multi-view graph autoencoder) emerged as the overall top performer, demonstrating superior robustness across almost all tasks and maintaining an F1 score of ~0.73 even in the difficult CV3 scenario.
Key Insights
- Complexity vs. Robustness: While DL models generally led in performance, traditional Matrix Factorization models like GRSMF proved remarkably stable, particularly as data imbalance increased.
- The Importance of Negative Sampling: The method used to select "non-SL" pairs significantly impacts model training. Incorporating gene expression data (NSM_Exp) to select negatives improved classification performance, highlighting the value of high-quality negative data.
- Ranking vs. Classification: Models generally performed better at binary classification (Is this pair SL?) than at ranking (prioritizing the most likely SL partners for a specific gene), which is often the more relevant task for experimental prioritization.
Future Directions
The benchmark identifies three critical areas for future research: context-specificity (moving beyond "global" predictions to cell-line specific ones), interpretability (addressing the "black box" nature of DL), and the development of gold-standard negative datasets. By establishing a rigorous evaluation standard, this study provides a roadmap for developing more reliable computational tools for synthetic lethality discovery.
Summary updated