A comparison of deep learning and linear-nonlinear cascade approaches to neural encoding

Theodore Moskovitz, Nicholas Roy, Jonathan W. PillowbioRxiv 2018 · DOI

Abstract

A large body of work on neural encoding has focused on “cascade” type models such as the linear-nonlinear-Poisson (LNP) model. This approach seeks to describe the encoding process in terms of a series of stages: (1) projection of the stimulus onto a bank of linear filters; (2) a nonlinear function combining these filter outputs; and (3) a noisy spike generation process. Here we explore the relationship of the LNP modeling framework to more recent approaches arising from the deep learning literature. Specifically, we show that deep neural network (DNN) and convolutional neural network (CNN) models of neural activity sit firmly within the LNP framework, and correspond to particular parametrizations of the nonlinear stage of the LNP model. Using data from primate retina and primary visual cortex, we compare the performance of LNP models fit with deep learning methods to LNP models fit with traditional estimators, including spike-triggered covariance (STC), information-theoretic spike-triggered averaging and covariance (iSTAC), and maximum likelihood estimators also known as “maximally informative dimensions” (MID). We show that models with nonlinearities parametrized by deep networks achieve higher accuracy for a fixed number of filters, and can extract a larger number of informative filters than traditional models. Finally, we perform a dimensionality analysis of LNP models trained with deep learning methods, revealing that a large number of filters are needed to accurately describe the neural responses of many cells even early in the visual pathway. This result overturns one of the central tenets of the LNP modeling framework: that neural computations are low-dimensional, or depend on the stimulus only via its projection onto a small number of linear filters. We discuss the implications of these findings for both the fitting and interpretation of LNP encoding models.

Introduction

For decades, the Linear-Nonlinear-Poisson (LNP) model has served as the primary framework for understanding sensory processing in the brain. This elegant three-stage "cascade" describes how a stimulus is filtered, nonlinearly transformed, and converted into a probabilistic spike train. However, the rise of Deep Neural Networks (DNNs) in computer vision has raised questions about whether these "black box" models represent a fundamental departure from classical neuroscience models.

Research by Moskovitz et al. demonstrates that DNNs are not a replacement for the LNP framework, but rather a more flexible way to parameterize it. By treating DNNs as specific instances of LNP models, we can leverage deep learning to uncover complexities in neural responses that traditional methods often overlook.

The Neural Encoding Problem

The core challenge of neural encoding is determining how a neuron converts sensory input into electrical spikes. Traditional LNP models typically assume a low-dimensional bottleneck, positing that a neuron's response depends on only a few specific stimulus features (linear filters). For instance, a V1 neuron might be modeled as caring about only 7 or 8 edge orientations. Classical estimators like Spike-Triggered Covariance (STC) or Maximally Informative Dimensions (MID) were specifically designed to identify this small set of filters. However, these methods often struggle to capture the full richness of neural responses, especially under complex stimulus conditions.

Methods: Reframing DNNs as LNP Models

The authors reframe DNNs and Convolutional Neural Networks (CNNs) as specific implementations of the LNP framework. In this architecture:

  1. The first layer of the DNN functions as a bank of linear filters.
  2. Subsequent hidden layers act as a highly flexible, multi-dimensional nonlinearity ($f$).
  3. The output is fed into a Poisson process to predict spike counts ($r$).
Comparison of traditional LNP (a) and DNN-based LNP (b) architectures.

This "Deep LNP" approach was compared against traditional estimators, including iSTAC (information-theoretic), CBF (semi-parametric), and ES-pool (a hybrid excitatory/suppressive model). Testing was conducted on in vivo extracellular recordings from macaque retinal ganglion cells (RGCs) and primary visual cortex (V1) simple and complex cells.

Key Insights and Results

Challenging the Low-Dimensional Bottleneck

The most significant finding is the apparent "death" of the low-dimensional bottleneck theory. While traditional models saturate quickly as more filters are added, DNN-based models show continued performance improvements. This suggests that neurons, even early in the visual pathway, are sensitive to a much higher-dimensional subspace of the stimulus than the 2–8 filters typically assumed.

Performance and Filter Characteristics

  • Superior Accuracy: DNN-based models consistently outperformed traditional LNP estimators in predicting spike trains across both retina and V1 datasets.
  • Generalized Filters: DNNs learn "broader" filters compared to the narrow, "peaky" filters found by traditional methods, suggesting they capture a wider range of the stimulus space that still drives neural activity.
  • Hybrid Efficiency: Initializing a DNN’s first layer with filters from traditional methods like iSTAC can significantly reduce training time and improve final model performance.

Conclusion and Future Directions

While DNN-based LNP models offer superior predictive power, challenges remain. Predictive accuracy ($r^2$) in V1 is still lower than in the retina, likely due to inherent neural noise. Furthermore, the simple binary stimuli used in this study may not fully reveal the gap between deep learning and traditional models. Future research utilizing natural stimuli is expected to further highlight the advantages of the high-dimensional flexibility provided by deep neural networks, potentially mapping the true dimensionality of visual processing across the brain.

Summary updated

Abstract

Abstract A large body of work on neural encoding has focused on “cascade” type models such as the linear-nonlinear-Poisson (LNP) model. This approach seeks to describe the encoding process in terms of a series of stages: (1) projection of the stimulus onto a bank of linear filters; (2) a nonlinear function combining these filter outputs; and (3) a noisy spike generation process. Here we explore the relationship of the LNP modeling framework to more recent approaches arising from the deep learning literature. Specifically, we show that deep neural network (DNN) and convolutional neural network (CNN) models of neural activity sit firmly within the LNP framework, and correspond to particular parametrizations of the nonlinear stage of the LNP model. Using data from primate retina and primary visual cortex, we compare the performance of LNP models fit with deep learning methods to LNP models fit with traditional estimators, including spike-triggered covariance (STC), information-theoretic spike-triggered averaging and covariance (iSTAC), and maximum likelihood estimators also known as “maximally informative dimensions” (MID). We show that models with nonlinearities parametrized by deep networks achieve higher accuracy for a fixed number of filters, and can extract a larger number of informative filters than traditional models. Finally, we perform a dimensionality analysis of LNP models trained with deep learning methods, revealing that a large number of filters are needed to accurately describe the neural responses of many cells even early in the visual pathway. This result overturns one of the central tenets of the LNP modeling framework: that neural computations are low-dimensional, or depend on the stimulus only via its projection onto a small number of linear filters. We discuss the implications of these findings for both the fitting and interpretation of LNP encoding models.
BibTeX
@misc{moskovitz2018comparison,
  author = {Theodore Moskovitz and Nicholas Roy and Jonathan W. Pillow},
  title = {{A comparison of deep learning and linear-nonlinear cascade approaches to neural encoding}},
  year = {2018},
  doi = {10.1101/463422},
  howpublished = {bioRxiv},
}
Figure