Bridging big data in the ENIGMA consortium to combine non-equivalent cognitive measures
Eamonn Kennedy, Shashank Vadlamani, Hannah M. Lindsey, Pui‐Wa Lei, Mary Jo Pugh, Paul M. Thompson, David F. Tate, Frank G. Hillary, Emily L. Dennis, Elisabeth A. Wilde, Maheen M. Adamson, Martin Alda, Sílvia Alonso-Lana, Sonia Ambrogi, Tim Anderson, Celso Arango, Robert F. Asarnow, Mihai Avram, Rosa Ayesa‐Arriola, Talin Babikian, Nerisa Banaj, Laura Bird, Stefan Borgwardt, Amy Brodtmann, Katharina Brosch, Karen Caeyenberghs, Vince D. Calhoun, Nancy D. Chiaravalloti, David X. Cifu, Benedicto Crespo‐Facorro, John C. Dalrymple‐Alford, Kristen Dams-O’Connor, Udo Dannlowski, David Darby, Nicholas D. Davenport, John DeLuca, Covadonga M. Díaz‐Caneja, Seth G. Disner, Ekaterina Dobryakova, Stefan Ehrlich, Carrie Esopenko, Fabio Ferrarelli, Lea E. Frank, Carol E. Franz, Paola Fuentes‐Claramonte, Helen M. Genova, Christopher C. Giza, Janik Goltermann, Dominik Grotegerd, Marius Gruber, Alfonso Gutiérrez‐Zotes, Minji Ha, Jan Haavik, Charles H. Hinkin, Kristen R. Hoskinson, Daniela Hubl, Andrei Irimia, Andreas Jansen, Michael Kaess, Xiaojian Kang, Kimbra Kenney, Barbora Keřková, Mohamed Salah Khlif, Minah Kim, Jochen Kindler, Tilo Kircher, Karolína Knížková, Knut K. Kolskår, Denise Krch, William S. Kremen, Taylor Kuhn, Veena Kumari, Jun Soo Kwon, Roberto Langella, Sarah Laskowitz, Jungha Lee, Jean Lengenfelder, Spencer W. Liebel, Victoria Liou‐Johnson, Sara M. Lippa, Marianne Løvstad, Astri J. Lundervold, Cassandra Marotta, Craig A. Marquardt, Paulo Mattos, Ahmad Mayeli, Carrie R. McDonald, Susanne Meinert, Tracy R. Melzer, Jessica Merchán‐Naranjo, Chantal Michel, Rajendra A. Morey, Benson Mwangi, Daniel J. Myall, Igor Nenadić, Mary R. Newsome, Abraham Nunes, Terence J. O’Brien, Viola Oertel, John OllingerScientific2024 · DOI
Investigators in neuroscience have turned to Big Data to address replication and reliability issues by increasing sample size. These efforts unveil new questions about how to integrate data across distinct sources and instruments. The goal of this study was to link scores across common auditory verbal learning tasks (AVLTs). This international secondary analysis aggregated multisite raw data for AVLTs across 53 studies totaling 10,505 individuals. Using the ComBat-GAM algorithm, we isolated and removed the component of memory scores associated with site effects while preserving instrumental effects. After adjustment, a continuous item response theory model used multiple memory items of varying difficulty to estimate each individual's latent verbal learning ability on a single scale. Equivalent raw scores across AVLTs were then found by linking individuals through the ability scale. Harmonization reduced total cross-site score variance by 37% while preserving meaningful memory effects. Age had the largest impact on scores overall (- 11.4%), while race/ethnicity variable was not significant (p > 0.05). The resulting tools were validated on dually administered tests. The conversion tool is available online so researchers and clinicians can convert memory scores across instruments. This work demonstrates that global harmonization initiatives can address reproducibility challenges across the behavioral sciences.
Introduction
In the era of "Big Data," neuroscience researchers aggregate datasets from dozens of labs worldwide to achieve the statistical power necessary for reproducible results. However, these labs rarely use identical assessment tools. In cognitive science, this "Tower of Babel" problem is particularly acute for memory assessments. One clinic might use the California Verbal Learning Test (CVLT), while another uses the Rey Auditory Verbal Learning Test (RAVLT) or the Hopkins Verbal Learning Test-Revised (HVLT-R). These tests are non-equivalent, featuring different word list lengths and varying levels of difficulty. This discrepancy creates massive "site effects" that can mask critical biological signals, such as the effects of aging or traumatic brain injury (TBI).
Methods
The researchers employed a two-stage "mega-analysis" pipeline to harmonize raw individual data from 53 independent studies, totaling 10,505 individuals across 13 countries and 8 languages.
ComBat-GAM (Batch Harmonization): Originally developed for genomics, this algorithm was used to identify and remove spurious variations caused by different testing environments (site effects) while preserving biological variables like age and clinical status.
Continuous Item Response Theory (IRT): The team used a Continuous Response Model (CRM) to estimate a person’s "latent ability"—their true underlying memory capacity—by accounting for the specific difficulty and discriminative power of the test items.
This shared latent scale allowed for the creation of mathematical "crosswalks" to convert a score from one instrument into its equivalent on another.
Results
The analysis confirmed a significant difficulty gradient among the tests: the CVLT was found to be the most challenging, while the HVLT-R was the easiest. Harmonization proved essential for data integration, reducing total cross-site score variance by 37%.
Comparison of raw vs. harmonized and adjusted scores across CVLT, RAVLT, and HVLT-R.
Age as a Primary Driver: Age had the most significant impact on scores (an 11.4% effect), following a non-linear, quadratic decline.
Demographic Robustness: Once age and education were accounted for, race and ethnicity were not significant predictors of memory performance, suggesting the harmonized scale is robust across cultural backgrounds.
Model Validation: The harmonized conversion model showed a 9.5% lower root mean squared error (RMSE) compared to unadjusted models when validated against a group of individuals who were dually administered different tests.
Discussion and Limitations
The team has launched a public Verbal Learning Calculator to allow researchers to convert scores instantly. However, several limitations remain. The study used a binary classification for TBI, which does not capture the full spectrum of injury severity. Additionally, the data primarily represented Western, English-speaking populations, necessitating further validation in more diverse global cohorts. Finally, while recall trials harmonized well, "recognition" trials showed inconsistent behavior and could not be accurately converted using this model.
Conclusion
This work demonstrates that global harmonization initiatives can effectively address reproducibility challenges in the behavioral sciences. By creating a "Rosetta Stone" for cognitive data, the ENIGMA consortium has provided a framework for integrating massive, diverse datasets to better understand the human mind.
Summary updated
Abstract
Investigators in neuroscience have turned to Big Data to address replication and reliability issues by increasing sample size. These efforts unveil new questions about how to integrate data across distinct sources and instruments. The goal of this study was to link scores across common auditory verbal learning tasks (AVLTs). This international secondary analysis aggregated multisite raw data for AVLTs across 53 studies totaling 10,505 individuals. Using the ComBat-GAM algorithm, we isolated and removed the component of memory scores associated with site effects while preserving instrumental effects. After adjustment, a continuous item response theory model used multiple memory items of varying difficulty to estimate each individual's latent verbal learning ability on a single scale. Equivalent raw scores across AVLTs were then found by linking individuals through the ability scale. Harmonization reduced total cross-site score variance by 37% while preserving meaningful memory effects. Age had the largest impact on scores overall (- 11.4%), while race/ethnicity variable was not significant (p > 0.05). The resulting tools were validated on dually administered tests. The conversion tool is available online so researchers and clinicians can convert memory scores across instruments. This work demonstrates that global harmonization initiatives can address reproducibility challenges across the behavioral sciences.
BibTeX
@inproceedings{kennedy2024bridging,
author = {Eamonn Kennedy and Shashank Vadlamani and Hannah M. Lindsey and Pui‐Wa Lei and Mary Jo Pugh and Paul M. Thompson and David F. Tate and Frank G. Hillary and Emily L. Dennis and Elisabeth A. Wilde and Maheen M. Adamson and Martin Alda and Sílvia Alonso-Lana and Sonia Ambrogi and Tim Anderson and Celso Arango and Robert F. Asarnow and Mihai Avram and Rosa Ayesa‐Arriola and Talin Babikian and Nerisa Banaj and Laura Bird and Stefan Borgwardt and Amy Brodtmann and Katharina Brosch and Karen Caeyenberghs and Vince D. Calhoun and Nancy D. Chiaravalloti and David X. Cifu and Benedicto Crespo‐Facorro and John C. Dalrymple‐Alford and Kristen Dams-O’Connor and Udo Dannlowski and David Darby and Nicholas D. Davenport and John DeLuca and Covadonga M. Díaz‐Caneja and Seth G. Disner and Ekaterina Dobryakova and Stefan Ehrlich and Carrie Esopenko and Fabio Ferrarelli and Lea E. Frank and Carol E. Franz and Paola Fuentes‐Claramonte and Helen M. Genova and Christopher C. Giza and Janik Goltermann and Dominik Grotegerd and Marius Gruber and Alfonso Gutiérrez‐Zotes and Minji Ha and Jan Haavik and Charles H. Hinkin and Kristen R. Hoskinson and Daniela Hubl and Andrei Irimia and Andreas Jansen and Michael Kaess and Xiaojian Kang and Kimbra Kenney and Barbora Keřková and Mohamed Salah Khlif and Minah Kim and Jochen Kindler and Tilo Kircher and Karolína Knížková and Knut K. Kolskår and Denise Krch and William S. Kremen and Taylor Kuhn and Veena Kumari and Jun Soo Kwon and Roberto Langella and Sarah Laskowitz and Jungha Lee and Jean Lengenfelder and Spencer W. Liebel and Victoria Liou‐Johnson and Sara M. Lippa and Marianne Løvstad and Astri J. Lundervold and Cassandra Marotta and Craig A. Marquardt and Paulo Mattos and Ahmad Mayeli and Carrie R. McDonald and Susanne Meinert and Tracy R. Melzer and Jessica Merchán‐Naranjo and Chantal Michel and Rajendra A. Morey and Benson Mwangi and Daniel J. Myall and Igor Nenadić and Mary R. Newsome and Abraham Nunes and Terence J. O’Brien and Viola Oertel and John Ollinger},
title = {{Bridging big data in the ENIGMA consortium to combine non-equivalent cognitive measures}},
booktitle = {Scientific},
year = {2024},
doi = {10.1038/s41598-024-72968-x},
}