Network of Evolvable Neural Units: Evolving to Learn at a Synaptic Level

Paul Bertens, Seong-Whan LeearXiv 2019 · OpenReview

TL;DR

This paper introduces the Evolvable Neural Unit (ENU), a novel building block for neural networks that can learn biologically plausible neuron and synapse dynamics through evolutionary algorithms. By composing these ENUs into a network, the authors demonstrate a system capable of one-shot learning and reinforcement learning in a T-maze, showcasing a path towards more biologically inspired and adaptable AI.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have achieved remarkable success, but their underlying structure and learning mechanisms are highly abstracted from biological brains. While computational neuroscience offers more detailed models of neurons and synapses, these are often complex and difficult to integrate into scalable learning systems. The challenge is to bridge this gap: how can we create artificial neural networks that are not only powerful but also incorporate more biologically realistic and flexible processing mechanisms, allowing for more adaptive and efficient learning? Specifically, the authors aim to move beyond fixed, manually designed neuron and synapse models and instead evolve these components dynamically.

Methods

The core innovation is the Evolvable Neural Unit (ENU). Each ENU is essentially a small Recurrent Neural Network (RNN), specifically a Gated Recurrent Unit (GRU) with modifications. These ENUs are designed to learn and store "dynamic parameters" in their internal memory state, which can represent various aspects of neuronal or synaptic behavior (e.g., membrane potential, synaptic weight).

Key aspects of the ENU method:

  • Evolved Dynamics: Instead of pre-defining neuron and synapse functions (like Integrate-and-Fire or STDP), ENUs are evolved using Evolution Strategies (ES) to approximate these functions. This allows for flexibility and the discovery of novel dynamics.
  • Internal Memory: The ENU's memory state acts as a dynamic parameter store, allowing it to adapt its behavior over time based on inputs, mimicking biological plasticity.
  • Gating Mechanisms: Similar to GRUs, ENUs use reset, update, and cell gates to control how information is processed and stored in the memory state. An additional output gate is included for more fine-grained control and potential spike generation.
  • Network Composition: Multiple ENUs are combined to form a larger network. Crucially, the gate parameters of the ENUs are shared across the network (synapses and neurons), but each ENU maintains its own unique internal memory state. This shared parameter approach, combined with individual states, allows for scalable evolution.
  • Evolution Strategies (ES): ES is used for optimization because it can handle non-differentiable objective functions and is well-suited for evolving complex systems without requiring backpropagation through time.

Datasets

Not discussed in the paper. The paper focuses on evolving agents within simulated environments rather than training on pre-existing datasets.

Key Insights and Ideas

  • Evolving "How" to Learn: The ENU framework allows the network to evolve not just what to learn (e.g., a specific task), but how to learn by evolving the underlying dynamics of its components. This is a form of meta-learning or learning-to-learn.
  • Dynamic Parameters as Synaptic Weights: Synaptic strength and neuronal behavior are not fixed weights but are dynamically updated and stored in the internal memory of ENUs. This offers a more biologically plausible model of plasticity.
  • Emergent Spiking Behavior: The authors demonstrate that ENUs can evolve to mimic spiking neuron behavior (like Integrate-and-Fire) and synaptic plasticity (like STDP) without explicit programming of these rules, suggesting these dynamics can emerge from optimizing for a task.
  • One-Shot Learning Capability: The evolved T-maze agent exhibits one-shot learning, meaning it can adapt its behavior based on a single experience (e.g., encountering poison), which is a significant departure from the millions of samples often required by traditional deep reinforcement learning methods.
  • Biological Inspiration as a Design Principle: The work suggests that many complex biological mechanisms might be emergent properties of evolutionary processes optimizing for survival and learning, rather than strictly engineered features.

Key Results

  • Mimicking Biological Models: Single ENUs were successfully evolved to approximate the behavior of Integrate-and-Fire (IAF) neurons and neuromodulated Spike-Timing-Dependent Plasticity (STDP) rules.
  • For IAF, the evolved ENU accurately integrated graded input and produced spikes at the correct time, even with random input potentials.
  • For STDP, the ENU learned to update synaptic weights based on the timing difference between pre- and post-synaptic spikes, and crucially, only when a neuromodulatory signal was present.
  • Reinforcement Learning in T-Maze: A network composed of ENUs was evolved to solve a T-maze task.
  • The agent learned to navigate, find food, and avoid poison.
  • It demonstrated one-shot learning, adapting its behavior after a single negative reward experience.
  • The agent evolved to perform "exploratory" behavior to maximize rewards.
  • Interestingly, the output neurons of the agent evolved to exhibit spike-like patterns, even though this was not explicitly enforced by the objective function.

Limitations and Future Work

  • Computational Cost: Evolving complex networks of ENUs can be computationally intensive, although the authors highlight efficient GPU implementation.
  • Environment Specificity: While the T-maze task is a good demonstration, scaling to more complex environments and tasks will be a significant challenge.
  • Biological Realism vs. Necessity: It remains an open question which specific biological mechanisms are essential for intelligence and which are merely byproducts of evolution.
  • Future Directions:
    • Simulating and evolving entire cortical columns.
  • Scaling up the number of neurons and synaptic compartments.
  • Evolving separate ENUs for specific biological components like dendrites or axons.
  • Investigating open-ended evolution and artificial life to achieve higher-level intelligence.
  • Exploring the role of multiple channels and neurotransmitter-like signals for more flexible learning.

Summary updated

Abstract

Although Deep Neural Networks have seen great success in recent years through various changes in overall architectures and optimization strategies, their fundamental underlying design remains largely unchanged. Computational neuroscience on the other hand provides more biologically realistic models of neural processing mechanisms, but they are still high level abstractions of the actual experimentally observed behaviour. Here a model is proposed that bridges Neuroscience, Machine Learning and Evolutionary Algorithms to evolve individual soma and synaptic compartment models of neurons in a scalable manner. Instead of attempting to manually derive models for all the observed complexity and diversity in neural processing, we propose an Evolvable Neural Unit (ENU) that can approximate the function of each individual neuron and synapse. We demonstrate that this type of unit can be evolved to mimic Integrate-And-Fire neurons and synaptic Spike-Timing-Dependent Plasticity. Additionally, by constructing a new type of neural network where each synapse and neuron is such an evolvable neural unit, we show it is possible to evolve an agent capable of learning to solve a T-maze environment task. This network independently discovers spiking dynamics and reinforcement type learning rules, opening up a new path towards biologically inspired artificial intelligence.
BibTeX
@misc{bertens2019network,
  author = {Paul Bertens and Seong-Whan Lee},
  title = {{Network of Evolvable Neural Units: Evolving to Learn at a Synaptic Level}},
  year = {2019},
  eprint = {1912.07589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.07589},
}
Figure