Benchmarking machine learning methods for synthetic lethality prediction in cancer

Yimiao Feng, Yahui Long, He Wang, Yang Ouyang, Quan Li, Min Wu, Jie ZhengNature Communications 2024 · DOI

Abstract

Synthetic lethality (SL) is a type of genetic interaction that occurs when defects in two genes cause cell death, while a defect in a single gene does not. Targeting an SL partner of a gene mutated in cancer can selectively kill tumor cells. Traditional wet-lab experiments for SL screening are resource-intensive. Hence, many computational methods have been developed for virtual screening of SL gene pairs. This study benchmarks recent machine learning methods for SL prediction, including three matrix factorization and eight deep learning models. We scrutinize model performance using various data splitting scenarios, negative sample ratios, and negative sampling methods on both classification and ranking tasks to assess the models’ generalizability and robustness. Our benchmark analyzed performance differences among the models and emphasized the importance of data and real-world scenarios. Finally, we suggest future directions to improve machine learning methods for SL discovery in terms of predictive power and interpretability.

Introduction

In the quest for precision oncology, synthetic lethality (SL) has emerged as a promising strategy. By identifying gene pairs where the loss of both leads to cell death but the loss of either alone is tolerated, researchers can develop therapies that selectively target cancer cells harboring specific mutations. However, the sheer number of possible gene combinations—approximately 200 million in the human genome—makes exhaustive experimental screening impractical.

While numerous machine learning (ML) models have been proposed to predict SL interactions, they are often evaluated under idealized conditions that fail to reflect clinical or experimental reality. This study provides the first comprehensive benchmark of 11 state-of-the-art ML models, evaluating their performance across realistic, high-stakes scenarios.

The Benchmarking Framework

The researchers evaluated 11 models categorized into two main groups: Matrix Factorization (MF) and Deep Learning (DL), specifically Graph Neural Networks (GNNs). The benchmark was designed to test these models' generalizability, robustness, and sensitivity to data preparation strategies.

Overview of the benchmarking framework, including data collection, methods, and evaluation metrics.

The evaluation framework focused on three critical dimensions:

  • Generalizability (Data Splitting Methods - DSMs): Models were tested on three levels of difficulty: CV1 (both genes seen in training), CV2 (one gene unseen), and CV3 (neither gene seen in training, a "total cold start").
  • Robustness (Positive-to-Negative Ratios - PNRs): The ratio of known SL pairs to unknown pairs was varied from 1:1 to 1:50 to simulate the "needle in a haystack" nature of SL discovery.
  • Negative Sampling (NSMs): Since confirmed non-SL pairs are scarce, the study compared random selection against strategies informed by gene expression and dependency scores.

Data Integration

To power these models, the study integrated diverse biological datasets, including SynLethDB 2.0 for known SL interactions, BioGRID for protein-protein interaction networks, and functional annotations from Gene Ontology (GO) and KEGG. Additionally, data from the Cancer Dependency Map (DepMap) was used to refine negative sampling strategies through gene expression and CRISPR knockout dependency scores.

Performance Analysis and the "Cold Start" Challenge

A primary finding of the benchmark is the significant performance drop in "cold start" (CV3) scenarios. Most models appear to rely on the local network neighborhood of known genes rather than learning generalized biological principles of synthetic lethality.

Density plots of predicted scores for SLMGAE and GCATSL across CV1, CV2, and CV3 scenarios.

As shown in the score distributions, while models like SLMGAE maintain some separation between positive and negative samples even in CV3, others like GCATSL see their predictive power collapse, with scores for all sample types clustering together in a narrow spike. Despite this, SLMGAE (a multi-view graph autoencoder) emerged as the overall top performer, demonstrating superior robustness across almost all tasks and maintaining an F1 score of ~0.73 even in the difficult CV3 scenario.

Key Insights

  • Complexity vs. Robustness: While DL models generally led in performance, traditional Matrix Factorization models like GRSMF proved remarkably stable, particularly as data imbalance increased.
  • The Importance of Negative Sampling: The method used to select "non-SL" pairs significantly impacts model training. Incorporating gene expression data (NSM_Exp) to select negatives improved classification performance, highlighting the value of high-quality negative data.
  • Ranking vs. Classification: Models generally performed better at binary classification (Is this pair SL?) than at ranking (prioritizing the most likely SL partners for a specific gene), which is often the more relevant task for experimental prioritization.

Future Directions

The benchmark identifies three critical areas for future research: context-specificity (moving beyond "global" predictions to cell-line specific ones), interpretability (addressing the "black box" nature of DL), and the development of gold-standard negative datasets. By establishing a rigorous evaluation standard, this study provides a roadmap for developing more reliable computational tools for synthetic lethality discovery.

Summary updated

Abstract

Synthetic lethality (SL) is a gold mine of anticancer drug targets, exposing cancer-specific dependencies of cellular survival. To complement resource-intensive experimental screening, many machine learning methods for SL prediction have emerged recently. However, a comprehensive benchmarking is lacking. This study systematically benchmarks 12 recent machine learning methods for SL prediction, assessing their performance across diverse data splitting scenarios, negative sample ratios, and negative sampling techniques, on both classification and ranking tasks. We observe that all the methods can perform significantly better by improving data quality, e.g., excluding computationally derived SLs from training and sampling negative labels based on gene expression. Among the methods, SLMGAE performs the best. Furthermore, the methods have limitations in realistic scenarios such as cold-start independent tests and context-specific SLs. These results, together with source code and datasets made freely available, provide guidance for selecting suitable methods and developing more powerful techniques for SL virtual screening.
BibTeX
@inproceedings{feng2024benchmarking,
  author = {Yimiao Feng and Yahui Long and He Wang and Yang Ouyang and Quan Li and Min Wu and Jie Zheng},
  title = {{Benchmarking machine learning methods for synthetic lethality prediction in cancer}},
  booktitle = {Nature Communications},
  year = {2024},
  doi = {10.1038/s41467-024-52900-7},
}
Figure