What constitutes understanding of ventral pathway function?

Charles Connor, Gabriel Kreiman, Carlos R Ponce, Carl Craver, Margaret Livingstone, Martin Schrimpf, Binxu WangOpenReview 2021 · OpenReview

TL;DR

This paper argues that understanding the ventral visual pathway requires moving beyond simple stimulus-response correlations to causal explanations. It proposes a framework for standardizing how we describe neuronal tuning and compare findings across different computational models and experimental approaches, emphasizing the need for interpretable and predictive models.

What Problem Does This Paper Address?

The paper tackles a fundamental challenge in neuroscience: how do we truly understand the function of the ventral visual pathway, the part of the brain responsible for recognizing objects and complex visual scenes? For decades, research has relied on presenting simple stimuli (like lines) to neurons and observing their responses (spikes). While this has revealed basic tuning properties (e.g., orientation selectivity), it creates a gap between the physical stimulus (pixels) and the experimenter's interpretation (stimulus spaces). This gap becomes more pronounced as we move up the visual hierarchy to more complex stimuli like faces, where multiple interpretations are possible. The authors question whether current methods, including those using sophisticated computational models like Convolutional Neural Networks (CNNs), truly explain neuronal function or merely provide correlative predictions. They highlight the need for a standardized approach to compare different models and experimental findings, moving towards a causal understanding of how neurons represent information.

Methods

The paper doesn't present a single novel algorithm but rather proposes a conceptual framework and outlines experimental strategies. It discusses the use of computational models, particularly Convolutional Neural Networks (CNNs), as tools to predict neuronal activity. CNNs are described as being able to learn representations of natural scenes and their hidden unit activations can be used to fit linear models of neuronal responses. The authors also touch upon generative models (like those used in DeepDream) for creating stimuli that maximally excite neurons. The core methodological contribution is the call for a standardized experimental framework that tests various approaches (including correlative and causal methods) on the same neural data and uses a common set of stimuli.

Datasets

Not discussed in the paper. The paper focuses on the conceptual framework and experimental design rather than specific datasets used for training or evaluation. It refers to existing studies that use natural images and stimuli designed to probe specific neuronal selectivities.

Key Insights and Ideas

  • Pixels vs. Spikes vs. Spaces: The paper distinguishes between observable "pixels" and "spikes" and the experimenter's "methods" and "spaces." The crucial insight is that while the pixel-to-spike relationship is causal, the space-to-spike relationship is often correlational, leading to potential misinterpretations.
  • The Limits of Simple Stimuli: While foundational, the use of simplified stimuli (like lines) in early neuroscience experiments has limitations when trying to understand how the brain processes complex, naturalistic scenes.
  • CNNs as Predictive Tools, Not Necessarily Explanations: CNNs can achieve high prediction accuracy for neuronal activity, but their internal workings (the "what convolution exactly?") are not always transparent, making them partial explanations at best.
  • The Need for Causal Understanding: The authors emphasize that correlative evidence alone is insufficient. Causal manipulations are necessary to truly understand neuronal function, though current causal experiments are limited in dimensionality.
  • Standardization is Key: A common experimental framework is needed to compare and stress-test competing hypotheses and models, fostering a more unified understanding of the ventral visual pathway.

Key Results

Not discussed in the paper. The paper is a conceptual piece outlining a research agenda and framework rather than reporting specific experimental results or quantitative findings. It references existing work that shows CNNs can account for up to 60% of neuronal variance in response to natural images.

Limitations and Future Work

Limitations: * Dimensionality of Causal Manipulations: The authors acknowledge that current causal experiments are limited in their ability to manipulate the high-dimensional nature of object vision. * Interpretability of CNNs: While CNNs are transparent in their mechanisms, their content (what specific visual attributes filters represent) is not always clear. * Incomplete Stimulus Spaces: Tuning functions based on a priori or a posteriori dimensions may not cover the full range of stimuli an organism encounters.

Future Work: * Develop a common experimental framework to test and compare various approaches (correlative, causal, computational models) on the same neural data. * Create more comprehensive dissections and visualizations of CNN models fitted to neural data. * Encourage researchers to explicitly hypothesize how stimulus spaces might arise in the brain. * Develop shared code repositories, stimulus datasets, and guidelines for falsifying theories. * The ultimate goal is to develop predictive models that not only replicate neuronal activity but also explain the underlying mechanisms and content transformations, leading to advancements in computer vision and potential applications in neuroscience and prosthetics.

Summary updated

Abstract

Scientific question: How can we use computational models to decipher and/or represent neuronal tuning properties, and standardize descriptions for comparisons across stimuli/tasks/species? Introduction. At the onset of visual neuroscience, first there was light, and then came explanations. Working to stimulate neurons in primary visual cortex (V1) in 1958, Hubel and Wiesel projected white light onto cat retinas using a modified ophthalmoscope and a slide projector. They had glass- and brass slides with drawings and cutouts, using them to shape light into simple geometric patterns. Among their many findings, they established V1 neurons showed higher activity to specifically placed line segments – lines optimized in their location, length/width, color, and their rotation. The simplicity of these stimuli allowed for straightforward interpretations, specifically that V1 neurons signal contour orientation1. Observations vs. interpretations. There were five components that made these experiments canonical, and have been included in most subsequent studies of visual neuroscience: 1) A physical stimulus (e.g., light patterns on a projection screen/computer monitor). 2) A generative method for producing the physical stimuli (e.g., lines drawn manually on slides, variables for a computer graphics library, vectors in a generative adversarial network). 3) An experimenter-labeled stimulus space (e.g., orientation, categories) with a metric to order/cluster the physical stimuli (e.g., angular distance, perceptual similarity). 4) Neuronal activity associated with each stimulus (e.g., spike rates). 5) A potential mechanism suggesting how those tuning functions could arise from earlier inputs (e.g., spatially aligned projections from neurons in the midbrain [lateral geniculate nucleus]). The first and fourth components are observables (we refer to these as pixels and spikes). The second and third components are fundamentally entangled with the experimenter’s theories and interpretations (we refer to these as methods and spaces). The linchpin observation is that in this experimental design, the relationship between pixels and spikes is causal, but the relationship between spaces and spikes is correlational. Theoretically, there can be alternative explanations implicit in any given stimulus space which also affect neuronal activity — in an experiment, the subject’s brain only has access to the physical stimuli, not to the meaning attached to it.
BibTeX
@misc{connor2021what,
  author = {Charles Connor and Gabriel Kreiman and Carlos R Ponce and Carl Craver and Margaret Livingstone and Martin Schrimpf and Binxu Wang},
  title = {{What constitutes understanding of ventral pathway function?}},
  year = {2021},
  url = {https://openreview.net/forum?id=I913Sp8V9MX},
  howpublished = {OpenReview},
}
Figure