Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Samuel A. Nastase, Ariel Goldstein, Uri HassonNeuroImage 2020 · DOI

TL;DR

Cognitive neuroscience has historically relied on highly controlled, artificial experiments that may not accurately reflect how the brain functions in the real world. This paper argues for a shift towards naturalistic paradigms, drawing inspiration from machine learning's success in building predictive models from complex, real-world data.

What Problem Does This Paper Address?

The field of cognitive neuroscience has largely operated under the assumption that by meticulously controlling experimental variables and filtering out "noise," we can uncover fundamental, generalizable principles of brain function. This approach, rooted in traditional experimental design, aims to isolate simple, interpretable rules that govern neural and behavioral processes. However, the authors contend that this reductionist strategy may be fundamentally flawed. The brain evolved to operate in a complex, multidimensional, and dynamic real world, where variables are inherently intertwined and non-linear. By artificially decoupling these variables, we risk creating models that are highly specific to the laboratory context and fail to generalize to ecological behavior. This "generalizability crisis" is a significant challenge for building truly predictive and explanatory models of the brain.

Methods

The paper doesn't introduce new experimental methods or algorithms. Instead, it advocates for a conceptual shift in how cognitive neuroscience research is conducted. The core "method" proposed is the primacy of naturalistic paradigms. This involves:

  • Formulating hypotheses based on ecological considerations: Understanding the actual problems the brain faces in the real world should guide research questions.
  • Sampling brain activity under representative contexts: Instead of artificial manipulations, researchers should collect data during naturalistic tasks and in ecologically relevant settings.
  • Using controlled experiments to characterize boundary conditions: Controlled experiments can still be valuable for defining the limits of ecological brain function, but they should not be the primary source of hypothesis generation.
  • Formalizing hypotheses as explicit, predictive models: These models should be capable of making quantitative predictions about neural activity under naturalistic conditions.
  • Leveraging machine learning insights: The success of machine learning in building predictive models from complex, real-world data serves as a model for how cognitive neuroscience could progress.

Datasets

Not discussed in the paper. The paper references the use of publicly shared naturalistic datasets and the development of new technologies for data acquisition, but does not detail specific datasets used for evaluation within the paper itself.

Key Insights and Ideas

  • The "Generalizability Crisis": Highly controlled experiments, while useful for isolating variables, may lead to models that are too specific to the lab and fail to explain real-world behavior.
  • The Brain as a "High-Dimensional Modeling/Control Organ": Evolution has shaped the brain to handle complex, multidimensional interactions in the environment, not to operate like a scientist isolating simple variables.
  • Critique of "Systematic Design": The traditional approach of artificially reducing variables can obscure the very interactions the brain is designed to manage.
  • Advocacy for "Representative Design": Inspired by Egon Brunswik, this approach emphasizes sampling stimuli and conditions in a way that reflects their natural distribution and covariance.
  • Machine Learning as a Paradigm Shift: The success of machine learning models in matching or exceeding human performance on complex tasks, by embracing real-world data and prioritizing predictive power over interpretability, offers a valuable lesson for cognitive neuroscience.
  • Naturalistic Stimuli Reveal Nuance: Studies using naturalistic stimuli (e.g., for face perception or language processing) often reveal more complex and dynamic neural representations than those derived from artificial stimuli.

Key Results

The paper does not present quantitative results from a specific experiment. Instead, it synthesizes findings from existing literature to support its arguments. Key qualitative findings highlighted include:

  • V1 Neurons in Naturalistic Contexts: Work by David et al. (2004) demonstrates that the spatiotemporal tuning of neurons in primary visual cortex (V1) differs significantly between naturalistic and non-naturalistic contexts.
  • Face Perception Dynamics: Studies show that face perception circuitry becomes more nuanced when presented with complex, naturalistic stimuli, especially in social contexts.
  • Machine Learning Model Performance: Deep convolutional neural networks trained on "in the wild" face images achieve human-level or superior performance in face recognition, outperforming models trained on tightly controlled subsets of data.

Limitations and Future Work

Limitations:

  • Naturalistic paradigms are not a panacea: The authors acknowledge that naturalistic paradigms are not trivial to implement or analyze.
  • Potential for reduced statistical power: Naturalistic paradigms may resemble observational research and could potentially reduce statistical power for low-frequency phenomena due to intercorrelations among variables.

Future Work:

  • Shift in hypothesis generation: Hypotheses should stem from ecological considerations and address real-world problems.
  • Embrace complex data and tools: Building an ecological research program requires increasingly rich data and advanced quantitative tools.
  • Leverage publicly shared datasets: These datasets can serve as benchmarks for model comparison.
  • Develop advanced neuroimaging technologies: Technologies like iEEG, fNIRS, HD-DOT, and wearable MEG promise higher-fidelity data.
  • Utilize virtual reality paradigms: Immersive VR can facilitate more naturalistic fMRI studies.
  • Advance behavioral data collection: Experience sampling and mobile sensing offer new windows into naturalistic behavior.
  • Adopt "late commitment" and "system identification" approaches: These analytical frameworks allow for theoretical assumptions to be relaxed during data collection and imposed later, facilitating the mapping of formal models to neural responses.
  • Focus on predictive modeling: Emphasizing accounting for variance in real-world contexts can help combat reductionism.
  • Revolutionize cognitive neuroscience: The authors are optimistic that adopting an ecological perspective will not only complement but revolutionize the field.

Summary updated

BibTeX
@article{nastase2020keep,
  author = {Samuel A. Nastase and Ariel Goldstein and Uri Hasson},
  title = {{Keep it real: rethinking the primacy of experimental control in cognitive neuroscience}},
  journal = {NeuroImage},
  year = {2020},
  doi = {10.1016/j.neuroimage.2020.117254},
}
Figure