CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods

Shantanu Jain, Constantina Bakolitsa, Steven E. Brenner, Predrag Radivojac, John Moult, Susanna Repo, Roger A. Hoskins, Gaia Andreoletti, Daniel Barsky, Ajithavalli Chellapan, Hoyin Chu, Navya Dabbiru, Naveen K. Kollipara, Melissa Ly, Andrew J. Neumann, Lipika R. Pal, Eric Odell, Gaurav Pandey, Robin C. Peters-Petrulewicz, Rajgopal Srinivasan, Stephen F. Yee, Sri Jyothsna Yeleswarapu, Maya Zuhl, Ogün Adebalı, Ayoti Patra, M Beer, Raghavendra Hosur, Jian Peng, Brady Bernard, M. Victoria Berry, Shengcheng Dong, Alan P. Boyle, Aashish N. Adhikari, Jingqi Chen, Zhiqiang Hu, Robert Wang, Yaqiong Wang, Max J. Miller, Yanran Wang, Yana Bromberg, Paola Turina, Emidio Capriotti, James J. Han, Kıvılcım Öztürk, Hannah Carter, Giulia Babbi, Samuele Bovo, Pietro Di Lena, Pier Luigi Martelli, Castrense Savojardo, Rita Casadio, Melissa Cline, Greet De Baets, Sandra Bonache, Orland Dı́ez, Sara Gutiérrez‐Enríquez, Alejandro Fernández, Gemma Montalban, Lars Ootes, Selen Özkan, Natàlia Padilla, Casandra Riera, Xavier de la Cruz, Mark Diekhans, Peter J. Huwe, Qiong Wei, Qifang Xu, Roland L. Dunbrack, Valer Gotea, Laura Elnitski, Gennady Margolin, Piero Fariselli, Ivan V. Kulakovskiy, Vsevolod J. Makeev, Dmitry Penzar, Ilya E. Vorontsov, Alexander V. Favorov, Julia Forman, Marcia A. Hasenahuer, Marı́a Silvina Fornasari, Gustavo Parisi, Žiga Avsec, Muhammed Hasan Çelik, Thi Yen Duong Nguyen, Julien Gagneur, Fangyuan Shi, Matthew D. Edwards, Yuchun Guo, Kevin Tian, Haoyang Zeng, David K. Gifford, Jonathan Göke, Jan Zaucha, Julian Gough, Graham R. S. Ritchie, Adam Frankish, Jonathan M. Mudge, Jennifer Harrow, Erin L. Young, Yao YuGenome 2024 · DOI · OpenAlex

Abstract: Results show that while current methods are imperfect, they have major utility for research and clinical applications. Emerging methods and increasingly large, robust datasets for training and assessment promise further progress ahead.

Introduction

As DNA sequencing costs continue to plummet, the primary challenge in genomics has shifted from data acquisition to data interpretation. Every human genome contains millions of genetic variants; identifying the tiny fraction that are pathogenic (disease-causing) versus the vast majority that are benign is a monumental task. While laboratory experiments remain the gold standard for confirming a variant's effect, they are too resource-intensive to scale with the pace of sequencing.

Over the last decade, the Critical Assessment of Genome Interpretation (CAGI) has served as a community-driven "blind test" to rigorously evaluate the accuracy and clinical utility of computational tools designed to fill this gap. By moving away from self-reported success toward independent, blinded validation, CAGI provides an objective benchmark for the field.

The CAGI Framework and Methodology

CAGI follows a blind prediction model similar to the CASP experiments for protein structure. The process is divided into four distinct phases: challenge development, prediction season, assessment season, and dissemination. Data providers contribute unpublished genetic and phenotypic data, which participants then use to predict outcomes—such as enzyme activity or disease status—without knowing the true results. Finally, independent experts evaluate these submissions using standardized metrics.

Overview of CAGI challenges, participation, and data types across five editions, highlighting the growth in community involvement and the diversity of genomic data addressed.

A significant methodological advancement introduced through CAGI is the application of a Bayesian framework to map computational scores directly to clinical evidence levels (Supporting, Moderate, Strong, Very Strong) as defined by the ACMG/AMP guidelines. This allows clinicians to understand the specific weight a software prediction should carry in a medical diagnosis.

Diverse Datasets and Challenges

The paper reviews 50 challenges across five editions of CAGI, encompassing 738 submissions. These challenges cover a broad spectrum of genomic interpretation:

  • Biochemical Assays: Measuring the impact of missense variants on protein function (e.g., NAGLU and PTEN).
  • Clinical Databases: Using new entries in ClinVar and HGMD to evaluate tools prospectively and avoid training data contamination.
  • Regulatory Elements: Assessing the impact of mutations in non-coding regions, such as splicing and expression regulation.
  • Clinical Cases: Solving undiagnosed patient cases from institutions like SickKids and Johns Hopkins.
  • Complex Traits: Predicting risk for polygenic conditions like Crohn’s disease and bipolar disorder.

Key Insights and Clinical Utility

One of the most critical findings is the importance of the "left side" of the ROC curve. In clinical diagnostics, false positives are highly detrimental. CAGI emphasizes truncated ROC AUC, which measures performance in the high-confidence region. Many tools demonstrate exceptional ability to identify a subset of variants with very high certainty, even if their overall accuracy across all variants is lower.

The Bayesian framework maps predicted activity scores to the probability of pathogenicity, allowing computational results to meet ACMG/AMP clinical evidence thresholds.

In the NAGLU gene challenge, for instance, predictions were accurate enough that 6% of variants reached the "Strong" evidence threshold, and 31% met the "Supporting" threshold. Furthermore, in the SickKids challenges, CAGI participants successfully identified diagnostic variants that standard clinical pipelines had initially missed, demonstrating the power of community-sourced intelligence in solving complex medical mysteries.

Performance Results

  • Pathogenicity Prediction: Top-performing methods, such as REVEL and VEST, achieved Area Under the Curve (AUC) values between 0.85 and 0.92 for rare disease variants.
  • Biochemical Correlation: For missense variants, computational methods showed strong correlations with experimental enzyme activity, with Pearson’s r reaching up to 0.84.
  • Complex Traits: In predicting Crohn’s disease risk, the best models achieved an AUC of 0.72, providing a 20-fold range in relative risk—a level of discrimination sufficient to support clinical risk stratification.

Limitations and Future Directions

Despite significant progress, several challenges remain. The accuracy of computational tools is often capped by the inherent noise in the experimental data used for validation. Additionally, most current methods evaluate single mutations in isolation, whereas real-world disease often involves complex combinatorial effects (epistasis).

Future CAGI editions will focus on expanding data diversity to include non-European populations and more complex non-coding regulatory mechanisms. The authors anticipate that the recent "AlphaFold moment" in structural biology will soon catalyze a similar leap in the power and precision of variant interpretation tools, further bridging the gap between genomic data and clinical action.

Summary updated

Abstract

Results show that while current methods are imperfect, they have major utility for research and clinical applications. Emerging methods and increasingly large, robust datasets for training and assessment promise further progress ahead.
BibTeX
@inproceedings{jain2024cagi,
  author = {Shantanu Jain and Constantina Bakolitsa and Steven E. Brenner and Predrag Radivojac and John Moult and Susanna Repo and Roger A. Hoskins and Gaia Andreoletti and Daniel Barsky and Ajithavalli Chellapan and Hoyin Chu and Navya Dabbiru and Naveen K. Kollipara and Melissa Ly and Andrew J. Neumann and Lipika R. Pal and Eric Odell and Gaurav Pandey and Robin C. Peters-Petrulewicz and Rajgopal Srinivasan and Stephen F. Yee and Sri Jyothsna Yeleswarapu and Maya Zuhl and Ogün Adebalı and Ayoti Patra and M Beer and Raghavendra Hosur and Jian Peng and Brady Bernard and M. Victoria Berry and Shengcheng Dong and Alan P. Boyle and Aashish N. Adhikari and Jingqi Chen and Zhiqiang Hu and Robert Wang and Yaqiong Wang and Max J. Miller and Yanran Wang and Yana Bromberg and Paola Turina and Emidio Capriotti and James J. Han and Kıvılcım Öztürk and Hannah Carter and Giulia Babbi and Samuele Bovo and Pietro Di Lena and Pier Luigi Martelli and Castrense Savojardo and Rita Casadio and Melissa Cline and Greet De Baets and Sandra Bonache and Orland Dı́ez and Sara Gutiérrez‐Enríquez and Alejandro Fernández and Gemma Montalban and Lars Ootes and Selen Özkan and Natàlia Padilla and Casandra Riera and Xavier de la Cruz and Mark Diekhans and Peter J. Huwe and Qiong Wei and Qifang Xu and Roland L. Dunbrack and Valer Gotea and Laura Elnitski and Gennady Margolin and Piero Fariselli and Ivan V. Kulakovskiy and Vsevolod J. Makeev and Dmitry Penzar and Ilya E. Vorontsov and Alexander V. Favorov and Julia Forman and Marcia A. Hasenahuer and Marı́a Silvina Fornasari and Gustavo Parisi and Žiga Avsec and Muhammed Hasan Çelik and Thi Yen Duong Nguyen and Julien Gagneur and Fangyuan Shi and Matthew D. Edwards and Yuchun Guo and Kevin Tian and Haoyang Zeng and David K. Gifford and Jonathan Göke and Jan Zaucha and Julian Gough and Graham R. S. Ritchie and Adam Frankish and Jonathan M. Mudge and Jennifer Harrow and Erin L. Young and Yao Yu},
  title = {{CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods}},
  booktitle = {Genome},
  year = {2024},
  doi = {10.1186/s13059-023-03113-6},
}

Loading related content...

Figure