Interrogating theoretical models of neural computation with emergent property inference

Sean R. Bittner, Agostina Palmigiano, Alex T. Piet, Chunyu A. Duan, Carlos D. Brody, Kenneth D. Miller, John P. CunninghameLife 2021 · DOI

TL;DR

This paper introduces Emergent Property Inference (EPI), a novel deep learning-based method for inferring parameters of theoretical neural circuit models. EPI allows researchers to precisely control model behavior by conditioning on desired "emergent properties" (computational outcomes), offering a powerful new tool for understanding neural computation and revealing complex parametric structures.

What Problem Does This Paper Address?

Theoretical neuroscience often relies on building mathematical models (circuit models) to understand how neural circuits perform computations. A key challenge is the "inverse problem": given a desired computation or observed phenomenon, how do we determine the specific parameters (e.g., neuron conductances, synaptic strengths) of the model that produce it? Traditional methods either struggle with the complexity of biologically realistic models or lack precise control over the model's output. While some approaches focus on reproducing specific datasets, this paper argues that focusing on emergent properties – high-level computational outcomes – is more aligned with how theoretical neuroscience typically operates and offers deeper insights. The authors aim to bridge the gap between complex neural models and the need for efficient, insightful parameter inference.

Methods

The core contribution is Emergent Property Inference (EPI), a method that leverages deep neural networks and probabilistic modeling. EPI works by:

  1. Defining an Emergent Property (EP): This is a specific computational outcome of interest, quantified by statistics of the model's activity (e.g., firing rate, accuracy). It's defined by desired mean and variance constraints.
  2. Learning a Deep Probability Distribution: EPI uses a deep neural network (specifically, normalizing flows like Real NVP) to learn a probability distribution over model parameters. This distribution is constrained such that the model, when run with parameters sampled from it, reliably produces the defined emergent property.
  3. Optimizing with Augmented Lagrangian: The learning process uses an augmented Lagrangian method to optimize the parameters of the deep probability distribution, ensuring the emergent property constraints are met while maximizing the distribution's entropy (making it as general as possible).

This approach allows for:

  • Precise Control: Directly conditioning on the desired computational outcome.
  • Efficient Inference: Scaling better to high-dimensional parameter spaces compared to traditional methods like SMC-ABC.
  • Rich Analysis: The learned distribution can be analyzed to reveal parameter sensitivity and robustness, offering mechanistic insights.

Datasets

The paper does not use external datasets for training or evaluation. Instead, it uses simulated data generated by the theoretical models themselves. The "datasets" are essentially the outputs of these neural circuit models under various parameter configurations.

Key Insights and Ideas

  • Focusing on Emergent Properties is Powerful: By defining desired computational outcomes (emergent properties) rather than just raw data, EPI allows for more targeted and insightful parameter inference, aligning with the goals of theoretical neuroscience.
  • Deep Learning for Parameter Distributions: EPI harnesses the expressivity of deep probability distributions (normalizing flows) to capture complex, potentially multimodal, parameter distributions that traditional methods struggle with.
  • Parameter Sensitivity Revealed: The learned EPI distribution provides a local and global assessment of parameter sensitivity. This allows researchers to understand which parameters are critical for a specific computation and which are degenerate (less important).
  • Bridging Theory and Computation: EPI offers a principled way to use modern machine learning tools to interrogate and understand complex theoretical models of neural computation.

Key Results

  • Scalability: EPI demonstrated superior scalability in inferring high-dimensional RNN connectivities compared to SMC-ABC and SNPE, converging faster and with fewer simulations as the parameter dimension increased. For example, EPI was significantly more efficient than SNPE for RNNs with 50 or more neurons.
  • Stomatogastric Ganglion (STG): EPI successfully inferred parameters for a 5-neuron STG model that produced an "intermediate hub frequency," revealing complex parametric structure and sensitivity dimensions.
  • Primary Visual Cortex (V1): In a V1 model with multiple inhibitory subtypes, EPI uncovered a quadratic relationship between input variability and excitatory population variability, highlighting the importance of E- and P-population input and revealing parameter degeneracy.
  • Superior Colliculus (SC): EPI identified two distinct connectivity regimes in an SC model for rapid task switching. These regimes reproduced experimental findings from optogenetic inactivation experiments and revealed how different connectivity structures impact task performance and resilience.

Limitations and Future Work

  • Gradient Tractability: EPI relies on gradients of the emergent property. If the emergent property is computationally expensive or non-differentiable, EPI might not be suitable, and alternative methods like SNPE might be preferred.
  • Under/Over-constrained Problems: If the emergent property is not well-defined (too few constraints), the distribution might become unstable. If it's over-constrained (no parameter set can satisfy it), optimization will fail.
  • Future Work: The authors suggest that EPI can be applied to a wider range of theoretical models and that further exploration of its use in analyzing complex parametric structures, like those in the SC model, will yield deeper insights. They also highlight the potential for EPI to reveal links between network stabilization and compensatory interactions governing neural variability.

Summary updated

Abstract

A cornerstone of theoretical neuroscience is the circuit model: a system of equations that captures a hypothesized neural mechanism. Such models are valuable when they give rise to an experimentally observed phenomenon -- whether behavioral or a pattern of neural activity -- and thus can offer insights into neural computation. The operation of these circuits, like all models, critically depends on the choice of model parameters. A key step is then to identify the model parameters consistent with observed phenomena: to solve the inverse problem. In this work, we present a novel technique, emergent property inference (EPI), that brings the modern probabilistic modeling toolkit to theoretical neuroscience. When theorizing circuit models, theoreticians predominantly focus on reproducing computational properties rather than a particular dataset. Our method uses deep neural networks to learn parameter distributions with these computational properties. This methodology is introduced through a motivational example of parameter inference in the stomatogastric ganglion. EPI is then shown to allow precise control over the behavior of inferred parameters and to scale in parameter dimension better than alternative techniques. In the remainder of this work, we present novel theoretical findings in models of primary visual cortex and superior colliculus, which were gained through the examination of complex parametric structure captured by EPI. Beyond its scientific contribution, this work illustrates the variety of analyses possible once deep learning is harnessed towards solving theoretical inverse problems.
BibTeX
@inproceedings{bittner2021interrogating,
  author = {Sean R. Bittner and Agostina Palmigiano and Alex T. Piet and Chunyu A. Duan and Carlos D. Brody and Kenneth D. Miller and John P. Cunningham},
  title = {{Interrogating theoretical models of neural computation with emergent property inference}},
  booktitle = {eLife},
  year = {2021},
  doi = {10.7554/elife.56265},
}
Figure