Reliability of the NIH toolbox cognitive battery in children and adolescents: a 3-year longitudinal examination

Brittany K. Taylor, Michaela R. Frenzel, Jacob A. Eastman, Alex I. Wiesman, Yu‐Ping Wang, Vince D. Calhoun, Julia M. Stephen, Tony W. WilsonPsychological 2020 · DOI

The NIH Toolbox (NIH-TB) is widely considered the "gold standard" for large-scale neurodevelopmental research. However, a recent three-year longitudinal study reveals that its long-term reliability in children and adolescents is surprisingly low-to-moderate. These findings suggest that the battery currently falls short of the stability required for clinical use and barely meets the threshold for many research applications.

The Challenge of Longitudinal Stability

The NIH Toolbox was designed to be a "common currency" for researchers—a quick, computerized, and standardized way to measure cognitive, motor, and emotional function across the lifespan. It has been adopted by massive longitudinal projects like the Adolescent Brain and Cognitive Development (ABCD) study.

However, most existing reliability data for the NIH-TB comes from very short-term windows (7 to 21 days). In longitudinal studies, researchers test participants annually to track development. If the tool itself isn't stable over a year, it becomes impossible to distinguish between genuine neurodevelopmental change and measurement "noise." This study investigated whether the NIH-TB’s fully normed T-scores—adjusted for age, sex, ethnicity, and parental education—hold up over a three-year period in youth.

Scatter plots showing the correlation of fully normed T-scores across three years for various NIH-TB subtests, demonstrating the lack of perfect agreement over time.

Methodology

Researchers conducted a three-year longitudinal examination of 212 typically developing youth (ages 9–15) recruited across two sites: Omaha, NE, and Albuquerque, NM. Participants completed the NIH-TB Cognitive Battery annually. To assess reliability, the authors employed several statistical measures, including Intraclass Correlation Coefficients (ICCs), Concordance Correlation Coefficients (CCCs), and Bland-Altman Plots to visualize score drift.

Key Findings

The study utilized data from the Developmental Chronnecto-Genomics (Dev-CoG) study. The final analysis included 192 participants at Year 1, 162 at Year 2, and 118 at Year 3.

The Stability Gap

There is a stark contrast between the NIH-TB’s excellent short-term reliability and its mediocre long-term stability. This suggests the battery may be sensitive to transient factors like mood or fatigue, or that cognitive constructs in children are more volatile than the tool can currently capture.

"Standardized" is a Relative Term

Despite using the same computerized software, significant reliability differences emerged between the two study sites. This implies that subtle environmental factors or demographic differences can impact the tool's performance.

The Danger of Noise

In some tests, over 75% of the sample showed a score shift of more than one standard deviation (10 points) from one year to the next. In a clinical setting, a 10-point jump could be the difference between a "normal" classification and being flagged for cognitive impairment.

Bland-Altman plots illustrating the difference in scores between years, highlighting the measurement 'noise' and drift in individual performance.

Quantitative Results

  • Clinical Thresholds: For clinical use, a reliability coefficient of >0.90 is generally required. None of the NIH-TB subtests met this criterion.
  • Research Thresholds: For research, >0.70 is the standard. Only a few subtests (e.g., Oral Reading) and composite scores (e.g., Crystallized Cognition) consistently hit this mark.
  • Range of Reliability: ICCs for the full sample ranged from a low of 0.31 (Flanker test, Year 1 to 2) to a high of 0.76 (Pattern Comparison, Year 2 to 3).
  • Composite Superiority: Composite scores were generally more stable than individual subtests but still showed site-specific variability.

Limitations and Recommendations

The authors highlight several areas for caution:

  1. Sample Diversity: The cohort was predominantly Caucasian and native English-speaking. Current norming procedures might not be adequate for more diverse populations.
  2. Developmental Factors: This study focused only on youth. It remains unclear if these reliability issues are specific to rapid brain development or if they persist in adults.
  3. Comparison Tools: The study did not include traditional "paper-and-pencil" neuropsychological tests for a head-to-head comparison.

Conclusion: Researchers using NIH-TB data should interpret longitudinal results with caution. The authors call for a refinement of the battery’s norming procedures for children before it is adopted for clinical decision-making.

Summary updated

Abstract

Reliability of the NIH-TB Cognitive Battery was lower than expected given early work examining shorter test-retest intervals. Moreover, there were very few instances of tests meeting stability requirements for use in research; none of the tests exhibited adequate reliability for use in clinical applications. Reliability is paramount to establishing the validity of the tool, thus the constructs assessed by the NIH-TB may vary over time in youth. We recommend further refinement of the NIH-TB Cognitive Battery and its norming procedures for children before further adoption as a neuropsychological assessment. We also urge researchers who have already employed the NIH-TB in their studies to interpret their results with caution.
BibTeX
@inproceedings{taylor2020reliability,
  author = {Brittany K. Taylor and Michaela R. Frenzel and Jacob A. Eastman and Alex I. Wiesman and Yu‐Ping Wang and Vince D. Calhoun and Julia M. Stephen and Tony W. Wilson},
  title = {{Reliability of the NIH toolbox cognitive battery in children and adolescents: a 3-year longitudinal examination}},
  booktitle = {Psychological},
  year = {2020},
  doi = {10.1017/s0033291720003487},
}
Figure