Training and inferring neural network function with multi-agent reinforcement learning

Matthew Chalk, Gašper Tkačik, Olivier MarrebioRxiv 2019 · DOI

TL;DR

This paper introduces a novel framework that unifies training and understanding neural circuits by casting them as multi-agent reinforcement learning (RL) systems. It allows for training recurrent neural networks to perform specific functions and, crucially, enables inferring the underlying function of existing neural networks from their observed activity, paving the way for predicting how these networks adapt to changing conditions.

What Problem Does This Paper Address?

Systems neuroscience aims to understand what functions neural circuits perform and how they implement them. Traditional "top-down" approaches hypothesize a function and then build an ideal model circuit to match observed neural activity. However, this requires a priori knowledge of the function, which is often unknown. Furthermore, multiple circuit implementations can achieve the same function, making direct comparison difficult. This paper tackles this challenge by proposing a framework that doesn't require pre-specifying the function, allowing for both training and inferring the functional objective of neural networks.

Methods

The core innovation is framing the optimization of recurrent neural networks as a multi-agent reinforcement learning (RL) problem. Each neuron is treated as an "agent" that learns to adjust its firing behavior (its "action") to drive the network towards "rewarded states" that fulfill a specific function. This is achieved by defining a reward function that quantifies the desirability of network states for a given task.

Key techniques include:

  • Multi-Agent RL for Training: Using RL algorithms to train recurrent neural networks to perform diverse functions, such as efficient sensory coding or motor control.
  • Inverse RL for Inference: Developing a method to infer the reward function (and thus the underlying function) of a recorded neural network solely from its observed dynamics. This is a significant departure from traditional methods that require a hypothesized function.
  • Entropy Regularization: Employing entropy-regularized RL to ensure that the learned policies are not deterministic and to handle the inherent stochasticity of neural activity. This also helps in making the inverse RL problem well-posed.
  • Parametric Approximations: For larger networks, the authors propose using parametric approximations for value functions to make computations tractable, allowing for the inference of reward functions in more complex systems.

Datasets

The paper uses simulated datasets generated from the models described within the paper. These include:

  • Agent Navigating a Maze: A standard benchmark for RL, used to illustrate the general principles of RL and inverse RL.
  • Simulated Recurrent Neural Networks: Networks of varying sizes (e.g., 8 neurons, 12 neurons) with different input structures (binary input, pixel arrays) and tasks (spiking patterns, efficient coding). These simulations are used to train networks, infer reward functions, and predict adaptations.

No external, real-world neural datasets are used for direct evaluation in this paper.

Key Insights and Ideas

  • Unified Framework for Training and Inference: The most significant insight is the creation of a single framework that can both train neural networks to perform functions and infer the functions of existing networks from their activity. This bridges the gap between understanding and engineering neural computation.
  • Neurons as "Agents": Conceptualizing individual neurons as agents optimizing their behavior within a larger network context provides a powerful lens for understanding emergent network properties and learning.
  • Function as Invariant: The paper posits that the underlying function of a neural circuit is more likely to be invariant across different contexts (e.g., cell death, changing stimulus statistics) than its specific dynamics. This provides a strong theoretical basis for predicting how networks adapt.
  • Inverse RL for Biological Discovery: The ability to infer a network's objective function from its activity opens up new avenues for discovering the computational goals of biological neural circuits, moving beyond simple correlation analysis.

Key Results

  • Successful Training of Networks: The framework successfully trained recurrent neural networks to perform specific tasks, such as generating precise spiking patterns based on input, with optimized tuning curves.
  • Accurate Reward Inference: Inverse RL was able to accurately infer the underlying reward functions from simulated network dynamics, even without prior assumptions about the reward's form. For example, in one simulation, the inferred reward correctly identified the target spiking counts for different inputs.
  • Predictive Power of Inferred Functions: The inferred reward functions were used to predict how neural network dynamics would adapt to changes in input statistics or network structure (e.g., cell death), and these predictions matched simulation results. For instance, altering input statistics led to the prediction and observation of population splitting into different cell types.
  • Generalization Across Models: The framework demonstrated its generality by being applicable to different network architectures (e.g., pairwise coupled networks) and tasks (e.g., efficient coding).

Limitations and Future Work

  • Computational Scalability: The exact algorithm's state-space complexity scales exponentially with the number of neurons, making it infeasible for large networks. The paper addresses this with parametric approximations, but this introduces the need for additional assumptions.
  • Data Requirements for Inference: Inferring the reward function accurately requires a sufficient amount of observed neural data. The paper shows that the correlation between true and inferred rewards increases with data samples.
  • Assumption of Optimality: The framework assumes that the neural network is performing its function optimally. Real biological networks might only be approximately optimal, which could lead to misleading results. Future work could explore frameworks that account for approximate optimality.
  • Parametric Assumptions for Reward: For large networks, inferring the reward function often requires assuming a parametric form (e.g., sparse, pairwise, or global reward). Testing the validity of these assumptions and how they affect predictions is an area for future research.
  • Relaxing Markov Assumption: The current model assumes Markov dynamics. Future work could explore relaxing this by incorporating spiking history or refractory periods, though this would increase computational complexity.
  • Real-World Data Application: While the framework is theoretically powerful, its application to real, noisy biological data and validation against experimental observations is a crucial next step.

Summary updated

Abstract

Abstract A central goal in systems neuroscience is to understand the functions performed by neural circuits. Previous top-down models addressed this question by comparing the behaviour of an ideal model circuit, optimised to perform a given function, with neural recordings. However, this requires guessing in advance what function is being performed, which may not be possible for many neural systems. To address this, we propose a new framework for optimising a recurrent network using multi-agent reinforcement learning (RL). In this framework, a reward function quantifies how desirable each state of the network is for performing a given function. Each neuron is treated as an ‘agent’, which optimises its responses so as to drive the network towards rewarded states. Three applications follow from this. First, one can use multi-agent RL algorithms to optimise a recurrent neural network to perform diverse functions (e.g. efficient sensory coding or motor control). Second, one could use inverse RL to infer the function of a recorded neural network from data. Third, the theory predicts how neural networks should adapt their dynamics to maintain the same function when the external environment or network structure changes. This could lead to theoretical predictions about how neural network dynamics adapt to deal with cell death and/or varying sensory stimulus statistics.
BibTeX
@misc{chalk2019training,
  author = {Matthew Chalk and Gašper Tkačik and Olivier Marre},
  title = {{Training and inferring neural network function with multi-agent reinforcement learning}},
  year = {2019},
  doi = {10.1101/598086},
  howpublished = {bioRxiv},
}
Figure