Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings

Young Joon Kim, Nora Brackbill, Ella Batty, JinHyung Lee, Catalin Mitelut, William L. Tong, E. J. Chichilnisky, Liam PaninskibioRxiv 2020 · DOI

TL;DR

This paper introduces a novel, efficient nonlinear decoding method that significantly improves the reconstruction of natural images from large-scale primate retinal recordings. By combining linear decoding for coarse features and a neural network for fine details, they achieve state-of-the-art performance, demonstrating the crucial role of nonlinear computations for understanding visual information processing in the retina.

What Problem Does This Paper Address?

Understanding how the brain interprets sensory information is a fundamental challenge in neuroscience. A key aspect of this is "neural decoding" – figuring out what a stimulus was based on neural activity. This is particularly important for developing brain-machine interfaces. While previous work has decoded visual stimuli from neural activity, most approaches have relied on simple linear models and modest numbers of neurons, often using artificial stimuli. This paper tackles the problem of decoding complex, natural images from a large population of neurons in the primate retina, a task that has remained a significant challenge due to the complexity of natural scenes and the sheer volume of neural data.

Methods

The authors propose a multi-stage decoding approach:

  1. Linear Decoding for Low-Pass Features: They use penalized linear regression (ridge and LASSO) to decode the "low-pass" components of the images. These are essentially smoothed versions of the original images, capturing the broader structure. This step is computationally efficient and effective for global features.
  2. Nonlinear Decoding for High-Pass Details: A spatially-restricted neural network is employed to decode the "high-pass" components – the finer details and edges that are lost in the low-pass version. This network is designed to be efficient by focusing on the most relevant neurons for each pixel.
  3. Combined Decoding: The outputs from the linear and nonlinear decoders are combined to create a more complete image reconstruction.
  4. Deblurring Network: A final convolutional neural network, trained using natural image priors, is applied to further refine the decoded image, reducing blurriness and enhancing perceptual quality.

This hierarchical approach leverages the strengths of both linear and nonlinear methods, reserving the more computationally intensive nonlinear techniques for the challenging high-pass details.

Datasets

The study utilizes two large-scale datasets of macaque retinal ganglion cell (RGC) spike responses to natural scene images. Each dataset contains responses from over 1000 simultaneously recorded RGC units to 10,000 unique natural images. These datasets were used for both training and independent validation of the decoding methods.

Key Insights and Ideas

  • Necessity of Nonlinearity for Fine Details: The most striking insight is that while linear methods are sufficient for decoding the coarse, low-pass features of natural images, nonlinear computations are absolutely essential for accurately reconstructing the high-pass details (edges, textures).
  • Temporal Correlations Matter: Spike temporal correlations (how spikes from a single neuron are timed over short intervals) are crucial for nonlinear decoding of high-pass features and for both linear and nonlinear decoding of low-pass features. This highlights the importance of the precise timing of neural events, not just their overall rate.
  • OFF Midget Cells are Key for High-Pass: The study identifies OFF midget RGCs as particularly important contributors to the improved performance of nonlinear decoding for high-pass information. These cells appear to encode fine visual details in a more nonlinear fashion than other RGC types.
  • Hierarchical Decoding is Effective: The proposed multi-stage approach, separating low-pass and high-pass decoding, proves more effective than attempting to decode the entire image with a single method.

Key Results

  • The proposed nonlinear decoder achieved significantly higher accuracy in reconstructing natural images compared to linear decoders.
  • For low-pass features, linear decoders achieved high test correlations (e.g., 0.977).
  • For high-pass features, the nonlinear decoder achieved a test correlation of 0.360, a substantial improvement over linear methods (0.283).
  • The combined decoder (linear low-pass + nonlinear high-pass) yielded a mean test correlation of 0.901 against true images.
  • A final deblurring step further improved the combined decoder's output, reaching a test correlation of 0.912 and enhancing perceptual quality (measured by SSIM).
  • The method demonstrated scalability, working with over 1000 simultaneously recorded neurons.

Limitations and Future Work

  • Incomplete Detail Reconstruction: Despite improvements, the neural network decoder still missed many spatial details from the true image. The authors acknowledge that peripheral retina recordings have lower spatial acuity, but suggest that further improvements in nonlinear decoding methods are possible.
  • Deblurring Challenges: The deblurring step is described as a challenging problem, combining aspects of super-resolution, deblurring, denoising, and inpainting. Current image restoration techniques may not fully capture these complexities.
  • Further Investigation of Nonlinearity: The authors suggest that future work could explore other sources of nonlinearity, such as nonlinear spatial interactions within RGCs or between different RGC types, which might further explain visual decoding mechanisms. They also plan to investigate the role of nonlinear interactions between ON and OFF cells in signaling edges.

Summary updated

Abstract

Abstract Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve upon existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from > 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons. Author summary Neural decoding is a fundamental problem in computational and statistical neuroscience. There is an enormous literature on this problem, applied to a wide variety of brain areas and nervous systems. Here we focus on the problem of decoding visual information from the retina. The bulk of previous work here has focused on simple linear decoders, applied to modest numbers of simultaneously recorded cells, to decode artificial stimuli. In contrast, here we develop a scalable nonlinear decoding method to decode natural images from the responses of over a thousand simultaneously recorded units, and show that this decoder significantly improves on the state of the art.
BibTeX
@misc{kim2020nonlinear,
  author = {Young Joon Kim and Nora Brackbill and Ella Batty and JinHyung Lee and Catalin Mitelut and William L. Tong and E. J. Chichilnisky and Liam Paninski},
  title = {{Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings}},
  year = {2020},
  doi = {10.1101/2020.09.07.285742},
  howpublished = {bioRxiv},
}
Figure