Natural image synthesis for the retina with variational information bottleneck representation

Babak Rahmani, Demetri Psaltis, Christophe MoserNeurIPS 2022 · OpenReview

Introduction

In the early visual system, high-dimensional natural stimuli are encoded into trains of neuronal spikes that transmit information to the brain to produce perception. But is all that visual detail actually necessary to explain the neuronal responses?

Researchers have developed a new probabilistic model, the Information Bottleneck Gaussian Process (IB-GP), that identifies the "minimal" visual information needed to trigger specific neural responses in the retina. By combining the Information Bottleneck framework with Gaussian Processes, the model not only predicts retinal activity with high accuracy but can also synthesize simplified images—like Gabor filters and Gaussian patterns—that effectively "trick" the retina into responding as if it were seeing a complex natural scene.

The Challenge of Visual Encoding

In the field of neuroprosthetics (like retinal implants), a major hurdle is determining exactly what kind of electrical or visual signal will produce a desired perception in the brain. The retina is a complex biological computer; it takes high-dimensional visual data (millions of pixels) and compresses it into a series of electrical spikes sent via the optic nerve.

If we can identify the "minimal information stimuli"—the simplest possible patterns that elicit the same neural responses as complex natural scenes—we can design more efficient visual prostheses and better understand how the early visual system computes information.

Methods: The IB-GP Model

The authors introduce the IB-GP model to jointly learn the stimuli and retina codes.

IB-GP Model Architecture

It relies on two key mathematical frameworks:

  1. Information Bottleneck (IB): This technique is used to find a "bottleneck" representation ($Z$) of an input ($X$) that discards as much irrelevant information as possible while retaining maximum predictive power for an output ($Y$, the neural spikes).
  2. Gaussian Process (GP) Prior: Standard IB models assume data points are independent. However, video frames and eye movements are highly correlated over time. The authors integrated a GP prior to account for these temporal dynamics, allowing the model to learn the "flow" of visual information.

The researchers also implemented a closed-loop image synthesis pipeline. Once the IB-GP model was trained to understand the relationship between images and spikes, they used it to "prune" original natural images. They optimized a synthesizer to create new, low-complexity images that would produce the same latent "bottleneck" variables as the original complex images.

Datasets

The model was validated using two distinct ex-vivo retinal datasets:

  • Natural Dataset: Natural scene images (jittered to simulate eye movements) paired with spike trains from nine Retinal Ganglion Cells (RGCs) in a salamander.
  • Brownian-movement Dataset: Images of disc-like shapes moving in Brownian motion, paired with spike trains from 91 rat RGCs.

Key Insights and Results

The IB-GP model successfully captures the dynamics of stimuli-RGCs in its latent space, yielding several important findings.

Model Performance Comparison
  • Predictive Accuracy and Efficiency: The IB-GP model performed competitively with state-of-the-art feedforward Convolutional Neural Networks (CNNs) in predicting spike responses, despite being a more constrained probabilistic model. It achieved high Pearson correlation with as few as two latent variables, whereas models without the temporal prior (IB-Disjoint) struggled at that level of compression.
  • Biological Emergence: When the model was asked to synthesize the "minimal" stimuli required to drive the retina, it didn't just produce random noise. Instead, it generated patterns that look remarkably like classic biological filters: Gabor-type filters (edges), ON-center Gaussians, and OFF-center Gaussians. The model "discovered" these fundamental building blocks of vision on its own.
  • Temporal Matters: Incorporating the GP prior was crucial. Without it, the model required significantly more latent variables to achieve the same predictive accuracy. The IB-GP model "lumps" information into fewer, more meaningful factors.
  • Manifold Mapping: Using T-SNE to visualize the latent space, the researchers found that the IB-GP model perfectly clustered the dynamics of the stimuli. For example, in a test set of 50 unique images, the model's bottleneck representation naturally formed 50 distinct clusters, showing it had captured the "essence" of each visual scene.
  • Synthesis Success: In the closed-loop experiment, the synthesized "minimal" images achieved a high correlation with the original neural responses. When the model was trained jointly with the neural data, the synthesized images were much more effective at driving the RGCs than images produced by simple dimensionality reduction alone.

Limitations and Future Work

  • The "White Noise" Failure: The authors noted that the IB-GP model fails to fit neural data elicited by white noise. This is because white noise lacks the temporal correlations the GP prior expects—a "failure mode" that actually confirms the model is specifically tuned for natural scene statistics.
  • Computational Cost: Calculating the covariances for Gaussian Processes is mathematically "expensive," making it difficult to scale the model to extremely long datasets without further optimization.
  • Kernel Selection: While the chosen kernel worked well for multi-scale dynamics, exploring other spatio-temporal kernels might further improve how the model "disentangles" different visual features.

This work moves us a step closer to "speaking the language" of the retina, potentially leading to retinal implants that provide clearer, more naturalistic vision by focusing only on the information that truly matters to the brain.

Summary updated

Abstract

In the early visual system, high dimensional natural stimuli are encoded into the trains of neuronal spikes that transmit the information to the brain to produce perception. However, is all the visual scene information required to explain the neuronal responses? In this work, we search for answers to this question by developing a joint model of the natural visual input and neuronal responses using the Information Bottleneck (IB) framework that can represent features of the input data into a few latent variables that play a role in the prediction of the outputs. The correlations between data samples acquired from published experiments on ex-vivo retinas are accounted for in the model by a Gaussian Process (GP) prior. The proposed IB-GP model performs competitively to the state-of-the-art feedforward convolutional networks in predicting spike responses to natural stimuli. Finally, the IB-GP model is used in a closed-loop iterative process to obtain reduced-complexity inputs that elicit responses as elicited by the original stimuli. We found three properties of the retina's IB-GP model. First, the reconstructed stimuli from the latent variables show robustness in spike prediction across models. Second, surprisingly the dynamics of the high-dimensional stimuli and RGCs' responses are very well represented in the embeddings of the IB-GP model. Third, the minimum stimuli consist of different patterns: Gabor-type locally high-frequency filters, on- and off-center Gaussians, or a mixture of both. Overall, this work demonstrates that the IB-GP model provides a principled approach for joint learning of the stimuli and retina codes, capturing dynamics of the stimuli-RGCs in the latent space which could help better understand the computation of the early visual system.
BibTeX
@inproceedings{rahmani2022natural,
  author = {Babak Rahmani and Demetri Psaltis and Christophe Moser},
  title = {{Natural image synthesis for the retina with variational information bottleneck representation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=FkPZGtTxXx6},
}
Figure