Learning low-dimensional generalizable natural features from retina using a U-net

Siwei Wang, Benjamin Hoshal, Elizabeth de Laittre, Thierry Mora, Michael Berry, Stephanie E. PalmerNeurIPS 2022

How does the biological visual system compress the complex, high-dimensional world into a manageable neural code? In sensory neuroscience, understanding how the retina "encodes" the world into spikes is well-studied, but "decoding" those spikes back into a visual scene remains notoriously difficult. Both the stimulus (natural movies) and the response (thousands of firing neurons) are incredibly high-dimensional.

While previous models using Convolutional Neural Networks (CNNs) have successfully mimicked retinal output, they often result in "black box" representations. This paper introduces a modified U-net architecture to decode natural movies from salamander retinal activity, discovering a low-dimensional, interpretable, and generalizable latent space that represents "time in the natural scene."

A Novel Encoder-Decoder Architecture

Model architecture, generalization, and intrinsic dimensionality

To find a universal way of representing how a scene evolves over time, the researchers employed an encoder-decoder architecture based on a U-net backbone.

  • The Encoder: A ResNet18-based network processes 500ms of retinal activity, represented as Peri-Stimulus Time Histograms (PSTHs).
  • Variational Sampling: Unlike a standard U-net, the skip connections are modified to act as variational sampling layers. This forces the network to map neural activity into a compressed, continuous latent space ($Z$).
  • The Decoder: A symmetric network samples from this latent space to reconstruct a movie frame 100ms into the future.
  • Perceptual Loss: To avoid the blurry images often caused by simple pixel-wise Mean Squared Error, the model uses a loss function based on a pre-trained VGG19 network. This encourages the model to prioritize biologically meaningful features like edges and textures.

Datasets and Generalizability

The study utilized electrophysiological recordings from 93 salamander retinal ganglion cells exposed to three distinct 20-second natural movies: an aquatic "Fish" scene, a "Water" scene with ripples, and a terrestrial "Leaf" scene.

A major breakthrough of this approach is its generalizability. An encoder trained exclusively on the "Fish" movie could successfully decode the "Leaf" movie. This suggests the retina isn't merely memorizing specific objects; rather, it has evolved to encode universal spatio-temporal statistics common to all natural scenes.

Key Insights and Decoding Performance

Natural scenes are often too complex to parameterize fully. The authors ingeniously used "time within the movie" as a proxy for the evolving features. If the model can decode which frame it is looking at with high precision, it has successfully captured the scene's essential features.

  • Low Intrinsic Dimensionality: Despite the high-dimensional input, retinal activity exhibits a surprisingly low intrinsic dimension (around 4 to 5). A latent space of just 10 dimensions was sufficient for high-fidelity decoding.
  • High-Resolution Decoding: The model achieved remarkable precision, discriminating between movie frames just 17ms apart. With a latent space dimension of 5, decoding accuracy exceeded 80% across all movies, nearing 95-98% at 10 dimensions.
  • Information Saturation: The mutual information between the latent space and "time" nearly saturated the full entropy of the movie clip, indicating the compressed representation captures almost everything about the scene's progression.
Separation of static textures and dynamic motion features

Furthermore, the researchers demonstrated Feature Synergy. Using Information Theory, they showed that the retina encodes static textures and dynamic motion (optic flow) synergistically. Processing both together allows the retina to represent the state of the scene more efficiently than if it processed them separately.

Limitations and Future Directions

While highly successful, the model relies on PSTHs, which average out "noise correlations" between neurons. In a biological brain, the specific timing and correlations between individual spikes might carry additional information. Additionally, the current predictive horizon is limited to 100ms into the future; extending this to 500ms or a full second remains a challenging next step. Finally, while the U-net is a powerful computational tool, future work will need to investigate how specific biological retinal cell types physically implement these multi-scale, synergistic features.

Summary updated

BibTeX
@inproceedings{wang2022learning,
  author = {Siwei Wang and Benjamin Hoshal and Elizabeth de Laittre and Thierry Mora and Michael Berry and Stephanie E. Palmer},
  title = {{Learning low-dimensional generalizable natural features from retina using a U-net}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure