Introducing Neuromodulation in Deep Neural Networks to Learn Adaptive Behaviours

Nicolas Vecoven, Damien Ernst, Guillaume Drion

Introduction

Deep Neural Networks (DNNs) are exceptionally powerful but often brittle, excelling at specific tasks while struggling to adapt when environments change unexpectedly. In contrast, biological organisms are masters of adaptation. This flexibility is driven by neuromodulation: chemical signals like dopamine or serotonin that do not just transmit data, but dynamically alter the intrinsic properties of neurons.

Researchers have developed a new architecture called the Neuro-Modulated Network (NMN) that mimics this biological process to achieve rapid "on-the-fly" adaptation in meta-reinforcement learning settings.

The NMN Architecture

The NMN architecture decouples an agent into two interacting components: a Main Network and a Neuromodulatory Network.

The NMN architecture (A) consists of a neuromodulatory network providing context to a main network. The detailed activation function diagram (B) shows how the signal $z$ modifies the slope and bias of the activation.

The Main Network is a feed-forward network that processes immediate sensory inputs to produce actions. The Neuromodulatory Network is a recurrent neural network (RNN) that processes the "context"—the history of states and rewards. Instead of updating weights, which is a slow process, the Neuromodulatory Network outputs a signal ($z$) that modifies the activation functions of the Main Network in real-time.

Specifically, it tunes the slope and bias of saturated Rectified Linear Units (sReLUs). Mathematically, the activation becomes: $$\sigma_{NMN}(x, z; w_s, w_b) = \sigma(z^T(xw_s + w_b))$$ where $w_s$ and $w_b$ are learned parameters governing scale and offset. This allows the neuromodulatory signal to effectively "reprogram" the main network's response to stimuli based on the current task context.

Comparison between a standard meta-RL agent (A) and an NMN agent (B). In the NMN, the RNN output modulates the MLP's internal activations rather than simply acting as an additional input vector.

Experimental Evaluation

The architecture was evaluated on three custom-designed navigation benchmarks featuring continuous action spaces and sparse rewards:

  • 1D Target: An agent must find a target on a line where its observation is shifted by a hidden, task-specific bias.
  • 2D Navigation with Wind: The agent navigates to a target while being pushed by a hidden "wind" vector, requiring it to deduce the wind direction from movement errors.
  • Two-Target Navigation: The agent must discover which of two targets is "good" through trial and error.

Key Insights and Results

  • Faster Convergence and Higher Performance: Across all benchmarks, NMNs learned significantly faster than standard RNN-based meta-learning approaches. In the 1D benchmark, NMNs achieved an expected reward of 4534, nearing the theoretical Bayesian optimal policy score of 4679.
  • Linear Scaling: Unlike Hypernetworks that generate all weights for another network (scaling quadratically), NMNs scale linearly with the number of neurons, making them computationally efficient.
  • Functional Inversion: The neuromodulatory signal can invert the slope of activation functions, effectively turning an "excitatory" neuron into an "inhibitory" one based purely on context.
  • Architecture Robustness: NMNs maintained high performance regardless of the main network's depth, whereas standard RNNs' performance fluctuated significantly based on the number of hidden layers.

Limitations and Future Work

While NMNs show great promise in Reinforcement Learning, future work should explore their application in supervised meta-learning, such as few-shot image classification. Additionally, investigating more complex, non-linear parametric activation functions or incorporating more specific biological constraints could further enhance the robustness of these adaptive behaviors.

Summary updated

BibTeX
@inproceedings{vecoven2020introducing,
  author = {Nicolas Vecoven and Damien Ernst and Guillaume Drion},
  title = {{Introducing Neuromodulation in Deep Neural Networks to Learn Adaptive Behaviours}},
  booktitle = {PLoS},
  year = {2020},
  doi = {10.1371/journal.pone.0227922},
}
Figure