Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture

Pawel Ladosz, Eseoghene Ben-Iwhiwhu, Jeffery Dick, Nicholas Ketz, Soheil Kolouri, Jeffrey L. Krichmar, Praveen K. Pilly, Andrea SoltoggioIEEE TNNLS 2019 · OpenReview

Introduction

Reinforcement Learning (RL) has seen significant success in Markovian environments where the current observation provides sufficient information for decision-making. However, real-world scenarios often present Partially Observable Markov Decision Processes (POMDPs), where agents must rely on memory to navigate environments with ambiguous stimuli. This paper introduces the Modulated Hebbian plus Q Network Architecture (MOHQA), a bio-inspired approach designed specifically to tackle "confounding POMDPs"—environments characterized by repetitive observations and sparse, distal rewards.

The Challenge of Confounding POMDPs

Traditional RL agents, even those equipped with recurrent neural networks like LSTMs, struggle in confounding POMDPs. These environments present two primary hurdles:

  1. Confounding Observations: Agents encounter identical visual inputs (e.g., generic hallways) at different spatial locations, making localization impossible from a single frame.
  2. Sparse and Distal Rewards: Critical decisions often occur long before a reward is received. In these cases, standard Temporal Difference (TD) error propagation is inefficient, as the reward signal is "diluted" across many non-informative intermediate states.

In such settings, the "signal" of a correct decision is often buried under the "noise" of repetitive wait states. Standard memory-based models often fail to associate the specific early action with the eventual reward because the history of observations is dominated by these non-distinct states.

The MOHQA Architecture

To address these limitations, the authors propose MOHQA, a hybrid architecture that combines the strengths of deep learning with biological learning principles. The system utilizes a shared convolutional feature extractor that feeds into two parallel heads:

1. The DQN Head

A standard Deep Q-Network (DQN) component handles low-level control and feature representation. It learns via backpropagation, optimizing for long-term cumulative reward.

2. The MOHN Head (Modulated Hebbian Network)

The core innovation is the Modulated Hebbian Network (MOHN), which implements a bio-inspired associative memory.

  • Eligibility Traces: MOHN maintains a "chemical-like" memory of recent neural activity, allowing it to bridge the temporal gap between actions and rewards.
  • Rare Correlations: Unlike standard eligibility traces that track all activity, MOHN focuses only on the most significant associations—specifically the top and bottom $\theta\%$ of correlations. This sparsity allows the network to ignore the "noise" of repetitive confounding states and focus on distinct decision points.
  • Asymmetrical Learning: Inspired by Spike-Timing-Dependent Plasticity (STDP), the Hebbian rule captures causal, directional relationships between observations and actions over time.
Figure 3: Overview of the MOHQA framework, including agent-environment interaction (A), the underlying Markov process (B), and the visual rendering of the CT-graph benchmark (C).

Experimental Setup and Benchmarks

The architecture was rigorously tested on two distinct benchmarks designed to highlight the difficulties of confounding POMDPs:

  • Configurable Tree Graph (CT-graph): A synthetic 2D environment where researchers can precisely control the level of "confounding" by adjusting the number of identical wait states and the depth of the decision tree.
  • MALMO (Minecraft): A complex 3D navigation task where an agent must find rewards at the end of identical-looking corridors. This environment tests the model's ability to handle high-dimensional visual inputs and long-range dependencies.

Key Results and Performance

MOHQA demonstrates a significant performance leap over state-of-the-art baselines, including A2C, AMRL, and QRDQN+LSTM.

Figure 1: Average reward over episodes for the (a) Complex confounding POMDP (CT-graph) and (b) MALMO environment, showing MOHQA's superior performance and sample efficiency.

Performance Gains

In the CT-graph benchmark, MOHQA outperformed the best-performing baselines by over 33%. Its ability to quickly associate distal rewards with early decisions allows it to converge much faster than traditional TD-based methods.

Robustness and Efficiency

The "rare correlation" mechanism proves highly effective against environmental noise. When 10% random noise was introduced to the inputs, MOHQA was the only architecture capable of learning the correct path within a small number of episodes. Furthermore, the MOHN head acts as a "shortcut" for learning; by suggesting successful actions early, it helps the DQN's feature extractor distinguish between visually identical states that lead to different outcomes—a process the authors call "mutual bootstrapping."

Limitations and Future Outlook

While MOHQA represents a breakthrough in solving confounding POMDPs, it faces practical challenges. The current implementation is approximately 66% slower per training step than a standard DQN, primarily due to the lack of optimization for Hebbian operations in modern deep learning libraries. Additionally, the architecture introduces new hyperparameters, such as the correlation threshold, which require careful tuning.

Future work will focus on optimizing these bio-inspired operations and testing the architecture's generality across a broader range of standard RL benchmarks, such as Atari games and complex robotic control tasks.

Conclusion

By integrating modulated Hebbian learning with deep reinforcement learning, MOHQA provides a robust solution for environments where traditional memory mechanisms falter. The use of sparse, rare correlations allows agents to filter out environmental noise and focus on the causal links between their actions and distant rewards, bridging a critical gap in current AI navigation and decision-making capabilities.

Summary updated

Abstract

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN’s results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.
BibTeX
@misc{ladosz2019deep,
  author = {Pawel Ladosz and Eseoghene Ben-Iwhiwhu and Jeffery Dick and Nicholas Ketz and Soheil Kolouri and Jeffrey L. Krichmar and Praveen K. Pilly and Andrea Soltoggio},
  title = {{Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture}},
  year = {2019},
  eprint = {1909.09902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.09902},
}
Figure