Crowdsourced assessment of common genetic contribution to predicting anti-TNF treatment response in rheumatoid arthritis
Solveig K. Sieberts, Fan Zhu, Javier Garcı́a-Garcı́a, Eli A. Stahl, Abhishek Pratap, Gaurav Pandey, Dimitrios A. Pappas, Daniel Aguilar, Bernat Anton, Jaume Bonet, Ridvan Eksi, Oriol Fornés, Emre Güney, Hongdong Li, Manuel Alejandro Marín-López, Bharat Panwar, Joan Planas-Iglesias, Daniel Poglayen, Jing Cui, André O. Falcão, Christine Suver, Bruce Hoff, Venkat S. K. Balagurusamy, Donna Dillenberger, Elias Chaibub Neto, Thea Norman, Tero Aittokallio, Muhammad Ammad-ud-din, Chloé‐Agathe Azencott, Víctor Bellón, Valentina Boeva, Kerstin Bunte, Himanshu Chheda, Lu Cheng, Jukka Corander, Michel Dumontier, Anna Goldenberg, Peddinti Gopalacharyulu, Mohsen Hajiloo, Daniel Hidru, Alok Jaiswal, Samuel Kaski, Beyrem Khalfaoui, Suleiman A. Khan, Eric R. Kramer, Pekka Marttinen, Aziz M. Mezlini, Bhuvan Molparia, Matti Pirinen, Janna Saarela, Matthias Samwald, Véronique Stoven, Hao Tang, Jing Tang, Ali Torkamani, Jean-Phillipe Vert, Bo Wang, Tao Wang, Krister Wennerberg, Nathan E. Wineinger, Guanghua Xiao, Yang Xie, Rae S. M. Yeung, Xiaowei Zhan, Cheng Zhao, Manuel Calaza, Haitham Elmarakeby, Lenwood S. Heath, Quan Long, Jonathan D. Moore, Stephen O. Opiyo, Richard S. Savage, Jun Zhu, Jeff Greenberg, Joel Kremer, Kaleb Michaud, Anne Barton, Marieke J. H. Coenen, Xavier Mariette, Corinne Miceli‐Richard, Nancy A. Shadick, Michael E. Weinblatt, Niek de Vries, Paul P. Tak, Daniëlle M. Gerlag, T. Huizinga, Fina Kurreeman, Cornelia F Allaart, S. Louis Bridges, Lindsey A. Criswell, Larry W. Moreland, Lars Klareskog, Saedís Saevarsdóttir, Leonid Padyukov, Peter K. Gregersen, Stephen Friend, Robert Plenge, Gustavo Stolovitzky, Baldo Oliva, Yuanfang GuanNature Communications2016 · DOI
Rheumatoid arthritis (RA) affects millions world-wide. While anti-TNF treatment is widely used to reduce disease progression, treatment fails in approximately one-third of patients. Currently, no biomarker exists that identifies non-responders before treatment. We performed a rigorous community-based assessment of the utility of Single Nucleotide Polymorphism (SNP) data for predicting anti-TNF treatment efficacy in RA patients in the context of a DREAM Challenge. An open challenge framework enabled the comparative evaluation of predictions developed by 73 research groups using the most comprehensive available data and covering a wide range of state-of-the-art modelling methodologies. Despite a significant genetic heritability estimate of treatment non-response trait ($h^2=0.18, P=0.02$), no significant genetic contribution to prediction accuracy is observed. Results formally confirm that SNP information does not significantly improve predictive performance relative to standard clinical traits, thereby justifying a refocusing of future efforts on collection of other data types.
Introduction
Rheumatoid arthritis is a debilitating autoimmune disease. While anti-TNF therapies (e.g., adalimumab or infliximab) are a gold standard, they are expensive and fail in about one-third of patients. Currently, clinicians lack a method to predict response before starting treatment, leading to a "trial and error" approach that risks irreversible joint damage. This study aimed to determine if common genetic variation—specifically SNPs—could serve as a predictive biomarker for personalizing RA care.
Methods and Study Design
The researchers utilized a DREAM Challenge framework, a crowdsourced competition designed to evaluate predictive models transparently and rigorously. This approach allowed for the simultaneous testing of thousands of models ranging from standard regressions to complex machine learning techniques like Gaussian Process Regression (GPR).
Overview of the DREAM Challenge framework and dataset partitioning.
As shown in the figure above, the challenge consisted of a Competitive Phase, where 73 teams submitted models, followed by a Collaborative Phase, where the top 8 teams joined organizers to refine methods and specifically isolate the "value-add" of genetic data.
Datasets
The study leveraged three primary cohorts:
Discovery/Training Set: 2,031 RA patients with 2.5 million imputed SNPs and clinical covariates (age, sex, baseline disease activity).
Leaderboard Set: 675 patients used for real-time feedback during the competition.
Validation Set: An entirely independent cohort of 591 patients from the CORRONA CERTAIN study to ensure results were not overfitted to the training data.
Key Results
The Heritability Paradox
The study estimated the heritability ($h^2$) of anti-TNF response at 0.18, meaning roughly 18% of the variation in response is genetic. However, despite this measurable component, 73 expert teams using state-of-the-art AI could not translate that signal into a useful clinical prediction.
Clinical Data as the Performance Ceiling
The best models relied almost entirely on clinical traits. Adding genetic data provided no statistically significant improvement ($P = 0.85$ for AUROC). The top models achieved an Area Under the Receiver Operating Characteristic curve (AUROC) of approximately 0.62. For context, a random guess is 0.5, and a clinically actionable predictor typically requires an AUROC of 0.75 or higher.
Ensemble Failure
Even "stacking" (combining the best models into one ensemble) failed to break the performance ceiling, confirming that the limitation was the informative capacity of the data itself, not the algorithms employed.
Conclusion
By crowdsourcing the analysis, the authors turned a negative result into a definitive scientific statement: common SNPs do not provide sufficient information to predict anti-TNF response in RA. This justifies a refocusing of future efforts toward more dynamic data types, such as proteomics, transcriptomics, or direct analysis of synovial tissue, to solve the prediction puzzle in rheumatoid arthritis.
Summary updated
Abstract
Rheumatoid arthritis (RA) affects millions world-wide. While anti-TNF treatment is widely used to reduce disease progression, treatment fails in ∼one-third of patients. No biomarker currently exists that identifies non-responders before treatment. A rigorous community-based assessment of the utility of SNP data for predicting anti-TNF treatment efficacy in RA patients was performed in the context of a DREAM Challenge (http://www.synapse.org/RA_Challenge). An open challenge framework enabled the comparative evaluation of predictions developed by 73 research groups using the most comprehensive available data and covering a wide range of state-of-the-art modelling methodologies. Despite a significant genetic heritability estimate of treatment non-response trait (h(2)=0.18, P value=0.02), no significant genetic contribution to prediction accuracy is observed. Results formally confirm the expectations of the rheumatology community that SNP information does not significantly improve predictive performance relative to standard clinical traits, thereby justifying a refocusing of future efforts on collection of other data.
BibTeX
@inproceedings{sieberts2016crowdsourced,
author = {Solveig K. Sieberts and Fan Zhu and Javier Garcı́a-Garcı́a and Eli A. Stahl and Abhishek Pratap and Gaurav Pandey and Dimitrios A. Pappas and Daniel Aguilar and Bernat Anton and Jaume Bonet and Ridvan Eksi and Oriol Fornés and Emre Güney and Hongdong Li and Manuel Alejandro Marín-López and Bharat Panwar and Joan Planas-Iglesias and Daniel Poglayen and Jing Cui and André O. Falcão and Christine Suver and Bruce Hoff and Venkat S. K. Balagurusamy and Donna Dillenberger and Elias Chaibub Neto and Thea Norman and Tero Aittokallio and Muhammad Ammad-ud-din and Chloé‐Agathe Azencott and Víctor Bellón and Valentina Boeva and Kerstin Bunte and Himanshu Chheda and Lu Cheng and Jukka Corander and Michel Dumontier and Anna Goldenberg and Peddinti Gopalacharyulu and Mohsen Hajiloo and Daniel Hidru and Alok Jaiswal and Samuel Kaski and Beyrem Khalfaoui and Suleiman A. Khan and Eric R. Kramer and Pekka Marttinen and Aziz M. Mezlini and Bhuvan Molparia and Matti Pirinen and Janna Saarela and Matthias Samwald and Véronique Stoven and Hao Tang and Jing Tang and Ali Torkamani and Jean-Phillipe Vert and Bo Wang and Tao Wang and Krister Wennerberg and Nathan E. Wineinger and Guanghua Xiao and Yang Xie and Rae S. M. Yeung and Xiaowei Zhan and Cheng Zhao and Manuel Calaza and Haitham Elmarakeby and Lenwood S. Heath and Quan Long and Jonathan D. Moore and Stephen O. Opiyo and Richard S. Savage and Jun Zhu and Jeff Greenberg and Joel Kremer and Kaleb Michaud and Anne Barton and Marieke J. H. Coenen and Xavier Mariette and Corinne Miceli‐Richard and Nancy A. Shadick and Michael E. Weinblatt and Niek de Vries and Paul P. Tak and Daniëlle M. Gerlag and T. Huizinga and Fina Kurreeman and Cornelia F Allaart and S. Louis Bridges and Lindsey A. Criswell and Larry W. Moreland and Lars Klareskog and Saedís Saevarsdóttir and Leonid Padyukov and Peter K. Gregersen and Stephen Friend and Robert Plenge and Gustavo Stolovitzky and Baldo Oliva and Yuanfang Guan},
title = {{Crowdsourced assessment of common genetic contribution to predicting anti-TNF treatment response in rheumatoid arthritis}},
booktitle = {Nature Communications},
year = {2016},
doi = {10.1038/ncomms12460},
}