Introduction
Reinforcement Learning (RL) has seen significant success in Markovian environments where the current observation provides sufficient information for decision-making. However, real-world scenarios often present Partially Observable Markov Decision Processes (POMDPs), where agents must rely on memory to navigate environments with ambiguous stimuli. This paper introduces the Modulated Hebbian plus Q Network Architecture (MOHQA), a bio-inspired approach designed specifically to tackle "confounding POMDPs"—environments characterized by repetitive observations and sparse, distal rewards.
The Challenge of Confounding POMDPs
Traditional RL agents, even those equipped with recurrent neural networks like LSTMs, struggle in confounding POMDPs. These environments present two primary hurdles:
- Confounding Observations: Agents encounter identical visual inputs (e.g., generic hallways) at different spatial locations, making localization impossible from a single frame.
- Sparse and Distal Rewards: Critical decisions often occur long before a reward is received. In these cases, standard Temporal Difference (TD) error propagation is inefficient, as the reward signal is "diluted" across many non-informative intermediate states.
In such settings, the "signal" of a correct decision is often buried under the "noise" of repetitive wait states. Standard memory-based models often fail to associate the specific early action with the eventual reward because the history of observations is dominated by these non-distinct states.
The MOHQA Architecture
To address these limitations, the authors propose MOHQA, a hybrid architecture that combines the strengths of deep learning with biological learning principles. The system utilizes a shared convolutional feature extractor that feeds into two parallel heads:
1. The DQN Head
A standard Deep Q-Network (DQN) component handles low-level control and feature representation. It learns via backpropagation, optimizing for long-term cumulative reward.
2. The MOHN Head (Modulated Hebbian Network)
The core innovation is the Modulated Hebbian Network (MOHN), which implements a bio-inspired associative memory.
- Eligibility Traces: MOHN maintains a "chemical-like" memory of recent neural activity, allowing it to bridge the temporal gap between actions and rewards.
- Rare Correlations: Unlike standard eligibility traces that track all activity, MOHN focuses only on the most significant associations—specifically the top and bottom $\theta\%$ of correlations. This sparsity allows the network to ignore the "noise" of repetitive confounding states and focus on distinct decision points.
- Asymmetrical Learning: Inspired by Spike-Timing-Dependent Plasticity (STDP), the Hebbian rule captures causal, directional relationships between observations and actions over time.
Experimental Setup and Benchmarks
The architecture was rigorously tested on two distinct benchmarks designed to highlight the difficulties of confounding POMDPs:
- Configurable Tree Graph (CT-graph): A synthetic 2D environment where researchers can precisely control the level of "confounding" by adjusting the number of identical wait states and the depth of the decision tree.
- MALMO (Minecraft): A complex 3D navigation task where an agent must find rewards at the end of identical-looking corridors. This environment tests the model's ability to handle high-dimensional visual inputs and long-range dependencies.
Key Results and Performance
MOHQA demonstrates a significant performance leap over state-of-the-art baselines, including A2C, AMRL, and QRDQN+LSTM.
Performance Gains
In the CT-graph benchmark, MOHQA outperformed the best-performing baselines by over 33%. Its ability to quickly associate distal rewards with early decisions allows it to converge much faster than traditional TD-based methods.
Robustness and Efficiency
The "rare correlation" mechanism proves highly effective against environmental noise. When 10% random noise was introduced to the inputs, MOHQA was the only architecture capable of learning the correct path within a small number of episodes. Furthermore, the MOHN head acts as a "shortcut" for learning; by suggesting successful actions early, it helps the DQN's feature extractor distinguish between visually identical states that lead to different outcomes—a process the authors call "mutual bootstrapping."
Limitations and Future Outlook
While MOHQA represents a breakthrough in solving confounding POMDPs, it faces practical challenges. The current implementation is approximately 66% slower per training step than a standard DQN, primarily due to the lack of optimization for Hebbian operations in modern deep learning libraries. Additionally, the architecture introduces new hyperparameters, such as the correlation threshold, which require careful tuning.
Future work will focus on optimizing these bio-inspired operations and testing the architecture's generality across a broader range of standard RL benchmarks, such as Atari games and complex robotic control tasks.
Conclusion
By integrating modulated Hebbian learning with deep reinforcement learning, MOHQA provides a robust solution for environments where traditional memory mechanisms falter. The use of sparse, rare correlations allows agents to filter out environmental noise and focus on the causal links between their actions and distant rewards, bridging a critical gap in current AI navigation and decision-making capabilities.
Summary updated