Getting Blood From a Stone: Improving Brain-Behavior Inferences Without Brain Data

David J. Halpern, Todd M. GureckisbioRxiv 2021 · DOI

TL;DR

Cognitive neuroscience studies often suffer from low statistical power due to small sample sizes. This paper proposes a cost-effective strategy: collecting more behavioral data from additional participants, even without neuroimaging data, can significantly improve the precision of inferences about brain-behavior relationships. This offers a valuable alternative to simply increasing expensive neuroimaging data collection.

What Problem Does This Paper Address?

The field of cognitive neuroscience is grappling with a "replication crisis" partly fueled by low statistical power in many studies. This means that even when effects are found, they might be unreliable or overestimated. A common solution is to increase sample sizes, but neuroimaging data (like fMRI) is notoriously expensive and time-consuming to collect. Researchers often use behavioral data to infer underlying cognitive processes that are then related to neural signals. However, the precision of these inferences is limited by two main sources of error: sampling error (due to small sample sizes) and estimation error (when cognitive variables are indirectly measured and thus imperfectly estimated). This paper asks: can we improve our understanding of brain-behavior links by leveraging additional, cheaper behavioral data instead of solely relying on more costly neuroimaging data?

Methods

The authors explore two main avenues for improving inferences:

  1. Reducing Sampling Error: They introduce and analyze an estimator for correlation (Anderson's $\hat{\rho}$) that can incorporate data from individuals who only provided behavioral measures, not neuroimaging data. This is a statistical technique for handling "missing data" where one variable (neuroimaging) is missing for a subset of participants.
  2. Reducing Estimation Error: They investigate how collecting more behavioral data can improve the estimation of cognitive model parameters. This is particularly relevant for "model-based" neuroimaging, where complex cognitive models are used to derive latent variables from behavior. They explore hierarchical modeling and Bayesian approaches as ways to pool information across participants and improve parameter estimates.

The core of their approach involves simulations to quantify the benefits of this additional behavioral data under various scenarios (different true correlations, sample sizes, and model fits). They also perform an analysis on a real dataset from the Human Connectome Project.

Datasets

The paper uses simulated data extensively for its analyses. For empirical validation, they analyze data from the Human Connectome Project (HCP), specifically focusing on relationships between total grey matter volume and performance on two tasks: the Variable Short Penn Line Orientation (VSPLOT_TC) and a two-back working memory task (WM Task 2bk Acc).

Key Insights and Ideas

  • Behavioral Data as a "Free Lunch": The most striking idea is that collecting additional behavioral data, even without corresponding neural data, can significantly boost the precision of brain-behavior inferences. This is because behavioral data can help constrain estimates of latent cognitive variables and improve the overall statistical power of the analysis.
  • Cost-Benefit Tradeoff: The paper explicitly frames this as a design decision. Collecting 40 behavioral subjects online can be significantly cheaper than acquiring data from one additional fMRI subject, making it a potentially more efficient use of resources.
  • Beyond Simple Correlation: The benefits extend beyond simple correlation analyses. For model-based neuroimaging, improved behavioral data can lead to better estimates of cognitive parameters, which in turn leads to more precise inferences about their relationship with neural signals.
  • Hierarchical Modeling is Key: Hierarchical Bayesian models are highlighted as a powerful framework for integrating data from different sources (neuroimaging and behavioral-only) and pooling information across participants to improve estimates.
  • Not a Panacea: While beneficial, the authors acknowledge that additional behavioral data is not always helpful, especially when true effect sizes and sample sizes are very small. In such cases, more neuroimaging data might still be necessary.

Key Results

  • Simulations show that Anderson's $\hat{\rho}$ estimator, which incorporates behavioral-only data, can reduce estimation error compared to standard Pearson's $r$ using only complete (neuroimaging + behavioral) data, especially at higher sample sizes.
  • The paper quantifies the trade-off: in some scenarios, collecting a modest number of additional behavioral subjects can improve precision as much as acquiring one additional neuroimaging subject, at a fraction of the cost.
  • For hierarchical models, adding behavioral-only subjects can increase the correlation between estimated cognitive parameters and true parameters, leading to more powerful inferences about brain-behavior relationships.
  • Analysis of the Human Connectome Project data showed that the Anderson estimator performed comparably or better than standard methods, particularly at higher sample sizes.

Limitations and Future Work

  • Exchangeability Assumption: The paper assumes that behavioral data collected with and without neuroimaging are "exchangeable" (i.e., come from the same underlying distribution). In practice, this might not always be true, and more complex models might be needed to account for modality differences. Future work could investigate the magnitude of these differences and develop methods to aggregate inferences across modalities.
  • Model Dependence: The benefits of improved behavioral data can depend on the specific cognitive model used and the experimental design.
  • Linear Model Assumption: The paper primarily focuses on linear relationships. Future work could explore the value of behavioral data in non-linear regression contexts.
  • Low Sample Size/Effect Size Regime: The authors note that additional behavioral data is less helpful in scenarios with very small sample sizes and small true effect sizes, which are precisely the conditions where power is most needed. Future statistical work is needed to develop estimators that better control variance in these low-sample, low-correlation regimes.
  • Practical Implementation: While the statistical frameworks are presented, more work is needed to translate these methods into widely adopted practices for cognitive neuroscientists.

Summary updated

Abstract

Abstract In recent years, the cognitive neuroscience literature has come under criticism for containing many low-powered studies, limiting the ability to make reliable statistical in-ferences. Typically, the suggestion for increasing power is to collect more data with neural signals. However, many studies in cognitive neuroscience use parameters estimated from be-havioral data in order to make inferences about neural signals (such as fMRI BOLD signal). In this paper, we explore how cognitive neuroscientists can learn more about their neuroimaging signal by collecting data on behavior alone . We demonstrate through simulation that knowing more about the marginal distribution of behavioral parameters can improve inferences about the mapping between cognitive processes and neural data. In realistic settings of the correlation between cognitive and neural parameters, additional behavioral data can lead to the same improvement in the precision of inferences more cheaply and easily than collecting additional data from subjects in a neuroimaging study. This means that when conducting a neuroimaging study, researchers now have two knobs to turn in a design analysis: the number of subjects collected in the scanner and the number of behavioral subjects collected outside the scanner (in the lab or online).
BibTeX
@misc{halpern2021getting,
  author = {David J. Halpern and Todd M. Gureckis},
  title = {{Getting Blood From a Stone: Improving Brain-Behavior Inferences Without Brain Data}},
  year = {2021},
  doi = {10.1101/2021.01.21.427334},
  howpublished = {bioRxiv},
}
Figure