TL;DR
This paper presents a comprehensive evaluation of computational tools designed to predict the functional impact of genetic variants in the STK11 gene, a key player in cancer development. By leveraging novel experimental data from the CAGI challenge, the study reveals that while many tools perform well, there's still room for improvement, particularly in predicting variants with complex effects. The findings pave the way for more accurate variant interpretation, crucial for clinical diagnosis and personalized medicine.
What Problem Does This Paper Address?
Interpreting the functional consequences of genetic variations is a major bottleneck in genomics. As next-generation sequencing becomes more widespread, we're identifying an ever-increasing number of genetic variants, many of which are classified as "variants of uncertain significance" (VUS). This uncertainty hinders their use in clinical diagnosis and understanding disease mechanisms. The STK11 gene, for instance, is frequently mutated in non-small cell lung cancer (NSCLC), and understanding how these mutations affect its kinase activity is vital for developing targeted therapies. This paper addresses the challenge of accurately predicting the functional impact (specifically, kinase activity) of STK11 variants using computational tools, and evaluating how well these tools perform against experimental data.
Methods
The study employed a multi-faceted approach:
- Experimental Assays: Novel functional data was generated for 28 STK11 variants found in NSCLC biopsies. This included:
- A luciferase reporter assay to measure the variant's effect on TP53 transcriptional activity (a downstream effect of STK11). This provided a continuous measure of activity.
- A gel-shift assay to assess the variant's ability to undergo autophosphorylation, classifying variants as either "wildtype-like" (WT-like) or "loss-of-function" (LoF).
- Computational Predictor Evaluation: Both participant-developed models (from the CAGI challenge) and publicly available tools were evaluated.
- Regression and Classification Tasks: Predictors were assessed on their ability to:
- Regress the continuous R-WT activity values.
- Classify variants into WT-like or LoF categories.
- Uncertainty Quantification: Bootstrap methods were used to estimate confidence intervals for performance metrics, providing a robust assessment of predictor reliability.
- Clinical Variant Classification: A point-based system, incorporating functional assay results, computational predictions, population data, and evidence from known pathogenic variants, was used to re-classify variants and reduce VUS categories.
Datasets
- STK11 Variant Dataset: A dataset of 28 STK11 variants (27 missense, 1 single amino acid deletion) identified in primary NSCLC biopsies with low allele frequency (<1%) in gnomAD. This dataset was generated specifically for the CAGI challenge and represents a novel benchmark for evaluating STK11 variant predictors.
- Training Data for Predictors: Various sources were used by the participating teams, including ClinVar, gnomAD, UniRef, Pfam, and UniProt. Specific details are provided in Table 4.
Key Insights and Ideas
- Experimental Replicates are Crucial: The inclusion of both biological and technical replicates in the experimental assays allowed for a more realistic estimation of the maximum achievable performance for computational predictors, accounting for inherent assay variability.
- Predictors are Approaching Assay Performance: Several top-performing predictors, including the best participant model (3Cnet) and the well-established REVEL tool, demonstrated performance levels comparable to the experimental assay itself in distinguishing LoF from WT-like variants.
- The "Difficult-to-Predict" Variant: The variant p.H202R emerged as a particularly challenging case. While assays showed conflicting results (LoF in luciferase, WT-like in gel-shift), predictors struggled to accurately capture its impact. This suggests that current tools may excel at predicting direct kinase activity but may miss variants that affect downstream interactions (like p53 binding) without significantly altering kinase function.
- Clinical Actionability is Enhanced: By integrating multiple lines of evidence, the study successfully re-classified 16 previously uncertain variants into clinically actionable categories (likely pathogenic or likely benign), demonstrating the power of combined evidence for variant interpretation.
Key Results
- Top Performer: The 3Cnet model, a participant-developed tool, achieved the highest overall performance across Pearson's correlation (0.78), Kendall's Tau (0.58), and AUC (0.93) on the primary evaluation set.
- Strong Baseline: The publicly available REVEL tool performed exceptionally well, achieving Pearson's correlation of 0.82, Kendall's Tau of 0.66, and AUC of 0.95, outperforming most participant models.
- Experimental-Max Benchmark: The Experimental-Max predictor, representing the assay's performance with replicates, achieved Pearson's correlation of 0.83, Kendall's Tau of 0.68, and AUC of 0.96, serving as a benchmark for predictor performance.
- Variant Classification Success: 16 out of 24 variants with uncertain clinical significance were moved to definitive LP or LB classifications.
- Predictor Correlation: Top-performing predictors showed high pairwise correlations, suggesting they often leverage similar features or approaches.
Limitations and Future Work
- Small Evaluation Set: The study acknowledges that the evaluation was performed on a relatively small set of 22 variants. This limits the generalizability of the findings to other STK11 variants, and a larger dataset would be beneficial for more robust conclusions.
- Indel Handling: Some publicly available tools did not predict on the single amino acid deletion (p.K84del), requiring imputation that could potentially bias results.
- p.H202R Enigma: The difficulty in predicting p.H202R's effect highlights the need for models that can capture complex variant impacts beyond direct kinase activity, potentially incorporating protein-protein interaction data.
- Future Directions: The authors suggest that further research with larger variant sets and more replicates is needed to thoroughly characterize predictor performance. The success of the STK11 challenge also encourages the application of similar rigorous evaluation frameworks to other genes and biological processes.
Summary updated