Abstract
Computational methods for assessing the likely impacts of mutations, known as variant effect predictors (VEPs), are widely used in the assessment and interpretation of human genetic variation, as well as in other applications like protein engineering. Many different VEPs have been released to date, and there is tremendous variability in their underlying algorithms and outputs, and in the ways in which the methodologies and predictions are shared. This leads to considerable challenges for end users in knowing which VEPs to use and how to use them. Here, to address these issues, we provide guidelines and recommendations for the release of novel VEPs. Emphasising open-source availability, transparent methodologies, clear variant effect score interpretations, standardised scales, accessible predictions, and rigorous training data disclosure, we aim to improve the usability and interpretability of VEPs, and promote their integration into analysis and evaluation pipelines. We also provide a large, categorised list of currently available VEPs, aiming to facilitate the discovery and encourage the usage of novel methods within the scientific community.
Introduction: The Need for Standardization
In the era of precision medicine, Variant Effect Predictors (VEPs) are essential for determining whether specific genetic mutations are benign or disease-driving. However, the current VEP ecosystem lacks consistency, leading to several critical challenges:
- Inconsistency in Implementation: Tools often utilize disparate scales, naming conventions, and reference genomes, complicating cross-tool comparisons.
- The "Black Box" Problem: Many predictors are hosted solely as web servers that may eventually become unavailable, or as closed-source code that precludes independent auditing.
- Data Circularity: This fundamental methodological flaw occurs when a predictor is trained and evaluated on the same database (e.g., ClinVar), leading to artificially inflated performance metrics. Without explicit disclosure of training variants, fair "head-to-head" comparisons are impossible.
Proposed Framework: A Gold Standard for VEP Release
To address these issues, the Atlas of Variant Effects (AVE) Alliance has synthesized a policy framework establishing a "gold standard" for VEP development and release.
Key recommendations include:
- Open Source & Containerization: Developers should provide open-source code via platforms like GitHub or HuggingFace and offer containerized versions (e.g., Docker, Apptainer) to ensure long-term reproducibility.
- Standardized Referencing: The use of MANE (Matched Annotation from NCBI and EMBL-EBI) transcripts is recommended as the default reference to minimize "isoform confusion" in clinical settings.
- Feature Transparency: Reports should include not only the final score but also the input features and their relative importance, along with descriptions of how runtime scales with input size.
- Calibration and Interpretation: Tools should ideally use a 0-to-1 scale (least-to-most damaging). Developers are advised against using clinical labels like "Likely Pathogenic" for raw scores to avoid confusing clinicians and patients.
Data and Benchmarking
While this framework does not introduce a new dataset, it emphasizes the rigorous use of existing resources:
- Training and Evaluation: Standard databases such as ClinVar and gnomAD remain central.
- Independent Benchmarks: The authors advocate for using Multiplexed Assays of Variant Effect (MAVEs) and benchmarks like ProteinGym. These experimental results are less susceptible to "contamination" from the training data of supervised models.
- Community Resource: A "living" repository of VEPs has been launched at varianteffect.org/veps, providing metadata on methodologies and access links.
Critical Considerations: Circularity and Longevity
A major focus of these guidelines is the "circularity trap." Even unsupervised models, such as those based on protein language models, can suffer from circularity if they are tuned using datasets that overlap with evaluation sets. Furthermore, the paper warns against over-reliance on web servers. A VEP's scientific contribution is tied to its longevity; code should be archived in permanent repositories like Zenodo with a DOI.
There is also a call to move beyond simple pathogenicity scores toward predicting specific mechanisms, such as Gain-of-Function versus Loss-of-Function, which are vital for clinical decision-making.
Expected Impact
Adherence to these guidelines is expected to:
- Eliminate Circularity: Full disclosure of training sets allows for the identification of variant-level and gene-level overlaps.
- Improve Interoperability: Adopting HGVS notation and GA4GH standards ensures that VEP outputs are immediately compatible with clinical pipelines.
- Increase Scientific Trust: Transparency in pre-processing and methodology allows the community to verify state-of-the-art claims.
Limitations and Future Directions
Providing pre-computed scores for all possible variants is often computationally prohibitive. In such cases, providing the executable method is a necessary alternative. While these guidelines primarily address protein-coding regions, extending these standards to non-coding regions (e.g., promoters, UTRs) remains a significant challenge. Ultimately, the success of these voluntary guidelines depends on their adoption by journals and peer reviewers as a standard requirement for publication.
Summary updated