Learning with sparse reward in a gap junction network inspired by the insect mushroom body

Tianqi Wei, Qinghai Guo, Barbara WebbPLOS Computational Biology 2024 · DOI

TL;DR

This paper introduces a novel "dynamic routing model" inspired by the insect mushroom body, which uses electrical gap junctions to create a resistive network for efficient reinforcement learning, particularly in scenarios with sparse rewards. The model demonstrates promising results in navigation and associative learning tasks, outperforming traditional Q-learning in challenging sparse reward settings.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often struggle with "sparse reward" problems, where positive feedback is infrequent and only received upon achieving a distant goal. This makes it difficult for agents to learn effective strategies, as most actions provide no immediate indication of progress. While biological systems, like insects, seem adept at learning from sparse rewards, current RL algorithms often require extensive exploration or carefully crafted reward structures. This paper aims to bridge this gap by proposing a biologically inspired model that leverages the properties of gap junctions, found in neural circuits like the insect mushroom body, to create a more efficient learning mechanism for sparse reward scenarios.

Methods

The core of the paper is the "dynamic routing model" (DRM). This model abstracts a neural circuit into a resistive network where:

  • State Nodes: Represent states of the environment.
  • Gap Junctions (Resistors): Connect state nodes, with their weights (conductance) learning to represent the probability and direction of state transitions. These connections are adaptive and strengthen with experience.
  • Current Flow: When a "target" state is set (by lowering its potential, analogous to receiving a reward), a current flows through the network. This current naturally finds the shortest path from the current state to the target state.
  • Action Nodes: Receive input from the current flow and are associated with specific actions. The strength of the connection between a state transition and an action node is also learned.

The learning process involves:

  1. Latent Learning: During exploration, the weights of the gap junctions are adjusted based on observed state transitions, effectively building a graph of the environment's causal structure.
  2. Target Setting: When a reward is received, the corresponding state node is "grounded" (connected to a low potential), establishing it as a target.
  3. Action Decision: The agent chooses actions based on the direction and strength of the current flow, which guides it towards the target state.

The model uses nodal analysis to calculate current flow and an Euler method for solving the differential equations governing weight updates.

Datasets

The paper evaluates the DRM on three benchmark RL tasks:

  • Taxi-v3 (OpenAI Gym): A discrete grid-world navigation task with a taxi picking up and dropping off passengers. This task is configured with a truly sparse reward (only at the final goal) and an infinite step limit per episode, making it significantly more challenging than standard RL benchmarks.
  • Voronoi World: A more complex navigation task with a continuous 2D space, irregular obstacles, and arbitrary start/end locations. This task is designed to be more realistic than grid-world environments.
  • Simple Associative Learning (Petri Dish Task): A simplified environment inspired by insect olfactory learning experiments, where an agent learns to associate odors with rewards. This task allows for direct comparison to biological learning paradigms.

The authors note that the RL tasks used are more complex than those typically employed for evaluating MB models.

Key Insights and Ideas

  • Gap Junctions as a Learning Substrate: The paper proposes that the electrical coupling via gap junctions in parallel neural fibers (like in the insect mushroom body) can form a resistive network capable of encoding task structure and facilitating learning.
  • Current Flow as a Navigation Signal: The flow of electrical current through this resistive network naturally identifies shortest paths to target states, providing a powerful mechanism for decision-making, especially in sparse reward settings.
  • Latent Learning of Causality: The model learns the causal relationships between states through experience, building a graph that represents the environment's dynamics without explicit reward signals during exploration.
  • Biological Plausibility: The model offers a potential explanation for how insect brains might solve sparse reward problems, drawing parallels between the DRM's components and the insect mushroom body's neural architecture.
  • Efficiency in Sparse Rewards: The DRM demonstrates superior performance compared to Q-learning in challenging sparse reward scenarios, suggesting its potential for more efficient learning in complex environments.

Key Results

  • Taxi-v3: The DRM converged quickly and achieved optimal results with a sparse reward setup (only at the final step, no intermediate rewards, infinite steps per episode), reaching an average reward of -5 within a few hundred episodes and 20,000 steps. In contrast, Q-learning failed to converge under the same conditions, exhibiting oscillating rewards and frequently hitting step limits. The DRM also achieved convergence in approximately 20 steps per episode.
  • Voronoi World: The DRM learned the topology of the environment and converged to optimal performance in about 250 episodes or 25,000 steps, achieving a reward of around -10. Q-learning again failed to converge. The model also successfully handled tasks with multiple reward sources.
  • Simple Associative Learning: The model qualitatively replicated associative learning behavior observed in fruit flies, demonstrating the ability to learn associations between odors and rewards, and showing a learning index comparable to experimental results.

Limitations and Future Work

  • Biological Plausibility of Gap Junction Plasticity: The paper acknowledges that direct evidence for adaptive, directional gap junction plasticity as proposed in the model is still debated and not definitively proven in all biological contexts.
  • All-to-All Connectivity Assumption: The model assumes that any state can potentially connect to any other state, which is more extensive than the known connectivity of insect mushroom body KCs. Relaxing this assumption could introduce "innate" constraints and improve biological plausibility.
  • One-Hot State Encoding: Currently, the model uses a one-hot encoding for states, which is less efficient than the sparse coding observed in the insect mushroom body. The authors suggest extending the model to handle sparse coding by modifying edge strengthening rules and target setting.
  • Continuous Tasks: While the model is mathematically described as continuous, initial evaluations were on discrete tasks. Future work will explore applying it to continuous tasks with modifications to action selection and sensory input processing.
  • Potential for Actor-Critic Framework: The DRM's ability to identify subgoals and guide exploration suggests its potential as a "critic" component within an actor-critic RL architecture.

Summary updated

BibTeX
@article{wei2024learning,
  author = {Tianqi Wei and Qinghai Guo and Barbara Webb},
  title = {{Learning with sparse reward in a gap junction network inspired by the insect mushroom body}},
  journal = {PLOS Computational Biology},
  year = {2024},
  doi = {10.1371/journal.pcbi.1012086},
}
Figure