Population encoding of stimulus features along the visual hierarchy

Luciano Dyballa, Andra M. Rudzite, Mahmood S. Hoseini, Mishek Thapa, Michael P. Stryker, Greg D. Field, Steven W. Zucker

TL;DR

This paper introduces a novel "encoding manifold" technique to visualize how neural populations represent visual stimuli. They discovered that the mouse retina discretely encodes features, while V1 represents them continuously, challenging previous assumptions. Interestingly, popular convolutional neural networks (CNNs) used to model vision more closely resemble the retina's discrete encoding than V1's continuous representation.

What Problem Does This Paper Address?

The visual system is organized hierarchically, with information processed through successive stages, from the retina to primary visual cortex (V1) and beyond. A fundamental question is how the neural populations at each stage partition the vast space of visual features. Do they organize information into distinct, specialized groups (like channels), or do they represent features in a more continuous, overlapping manner? Previous research has hinted at differences between the retina and V1, but a clear, quantitative comparison of their population-level encoding strategies has been lacking. This paper aims to bridge that gap by developing a method to visualize and compare these encoding strategies across different levels of the visual hierarchy and even in artificial neural networks.

Methods

The core innovation is the "encoding manifold" technique. Unlike previous "stimulus perspective" approaches that organize stimuli based on population responses, this method organizes neurons based on their individual responses to a diverse set of visual stimuli.

Here's how it works:

  1. Stimulus Presentation: Mouse retinal ganglion cells (RGCs) and V1 neurons were recorded using multi-electrode arrays while being presented with a battery of stimuli, including drifting gratings and flow patterns (mimicking naturalistic scenes).
  2. Response Measurement: Neural responses (spike trains) were recorded for each neuron to each stimulus.
  3. Manifold Construction:
    • Nonnegative Tensor Factorization (NTF): This was used to identify latent components that capture the relationship between stimuli and neural responses across the population.
  4. Neural Encoding Space: A matrix was constructed where each row represents a neuron and columns represent the "neural factors" derived from NTF. This effectively creates a "stimulus/response-space" for each neuron.
  5. Weighted Data Graph: A similarity kernel was applied to this space to build a graph where neurons with similar response profiles are connected.
  6. Diffusion Maps: This non-linear dimensionality reduction technique was applied to the data graph to infer the underlying manifold, where nearby points represent neurons with similar encoding properties.

This approach allows for a topological comparison of how different neural populations organize stimulus features.

Datasets

The study utilized two primary biological datasets:

  • Mouse Retina: Multi-electrode array (MEA) recordings from mouse retinal ganglion cells (RGCs) (n=1149).
  • Mouse Visual Cortex (V1): MEA recordings from mouse V1 neurons (n=640).

Additionally, they analyzed two popular convolutional neural networks (CNNs) trained on ImageNet:

  • ResNet50: A deep residual network.
  • VGG16: A widely used convolutional neural network architecture.

The stimuli used were designed to be isotropic (or repeating) to study response properties independent of receptive field location and included:

  • Drifting gratings at two spatial frequencies and eight directions.
  • Flow stimuli (dots and oriented line segments) at two contrasts (positive and negative) and eight directions.

Key Insights and Ideas

  • Retina vs. V1: Discrete vs. Continuous Encoding: The most striking finding is the fundamental difference in how the retina and V1 encode visual features. The retina's encoding manifold shows distinct clusters, suggesting discrete sampling of features. In contrast, V1's manifold is continuous, indicating a smooth, overlapping representation of features. This settles a long-standing debate about the role of "channels" in neural processing.
  • CNNs Mimic the Retina, Not V1: When applying the same encoding manifold analysis to popular CNNs, the researchers found that their encoding topology was much more clustered, resembling the retina's discrete organization rather than V1's continuous representation. This suggests that current CNNs, while good at classification, may not fully capture the way biological cortical circuits represent information.
  • Manifold as a Topological Map: The encoding manifold provides a powerful way to visualize the "shape" of neural representations. Neurons with similar response profiles are neighbors on the manifold, allowing researchers to trace pathways and understand how different stimulus features are represented and transformed.
  • Physiological Properties Align with Manifold Structure: In V1, the manifold revealed that physiological properties (like excitatory/inhibitory classification and laminar specificity) are well-organized and correlate with specific regions of the manifold, further validating the approach.

Key Results

  • Retinal Encoding: The encoding manifold for mouse RGCs showed clear, distinct clusters, indicating a discrete organization of stimulus feature representation. These clusters corresponded to known RGC types, confirmed by spatial receptive field mosaics.
  • V1 Encoding: The encoding manifold for mouse V1 neurons was continuous and smooth, without abrupt transitions between groups of neurons. This suggests a more distributed and overlapping representation of stimulus features.
  • CNN Encoding: Encoding manifolds for ResNet50 and VGG16 were highly clustered, even more so than the retina. These clusters often corresponded to feature maps within the CNN architecture.
  • Quantitative Comparison: While not explicitly stated with single accuracy numbers, the qualitative difference in topology (clustered vs. continuous) is the primary result, with CNNs being more clustered than the retina, and the retina being more clustered than V1.

Limitations and Future Work

  • Stimulus Set: The authors acknowledge that the choice of visual stimuli, while comprehensive, might not cover all possible features. They suggest future work could explore variations in contrast, luminance, depth, and chromatic tuning.
  • CNN Limitations: The study highlights a significant limitation of current CNNs in capturing the continuous encoding observed in V1. Future work could explore how architectural modifications (e.g., recurrence) or different training paradigms might lead to more V1-like encoding topologies.
  • Connecting Manifold to Anatomy: While the V1 manifold showed some correlation with laminar structure, a more detailed mapping between manifold positions and precise anatomical/connectomic features is a promising avenue for future research.
  • Broader Applicability: The authors believe the encoding manifold approach is generalizable to other sensory systems, gene networks, and high-dimensional datasets, suggesting its utility beyond visual neuroscience. They also propose investigating the dLGN and extrastriate areas to see if they exhibit retinal-like or V1-like encoding.

Summary updated

Abstract

The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.
BibTeX
@inproceedings{dyballa2024population,
  author = {Luciano Dyballa and Andra M. Rudzite and Mahmood S. Hoseini and Mishek Thapa and Michael P. Stryker and Greg D. Field and Steven W. Zucker},
  title = {{Population encoding of stimulus features along the visual hierarchy}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2024},
  doi = {10.1073/pnas.2317773121},
}

Loading related content...

Figure