Introduction
Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains like the Arcade Learning Environment (ALE). However, a persistent challenge remains: sample inefficiency. Standard algorithms such as Deep Q-Networks (DQN) often require hundreds of millions of environment interactions to converge. This inefficiency stems from the reliance on "slow" gradient-based optimization and one-step temporal difference (TD) learning, where rewards propagate backward through state-action chains incrementally.
In contrast, biological systems utilize episodic memory—often associated with the hippocampus—to rapidly encode and retrieve successful experiences. The Episodic Memory Deep Q-Network (EMDQN) bridges this gap by integrating a biologically-inspired episodic memory module with the generalization capabilities of deep neural networks.
Methodology: Integrating Memory and Generalization
EMDQN maintains a dual-system architecture: a parametric Q-network for generalization and a non-parametric memory table for rapid learning from high-reward experiences.
1. Architecture and Fast Lookup
The system utilizes a standard Convolutional Neural Network (CNN) to predict action values ($Q_\theta$). Simultaneously, it maintains an Episodic Memory Table ($H$) that stores the maximum Monte-Carlo return—the total actual reward from a state to the episode's end—ever encountered for specific state-action pairs. To ensure computational efficiency, high-dimensional state images are mapped to a low-dimensional space using a fixed Gaussian random projection, enabling rapid exact-match lookups without the overhead of K-Nearest Neighbors searches.
2. Dual-Target Loss Function
The primary innovation of EMDQN is its loss function, which supervises the Q-network using two distinct targets: $$L = \alpha(Q_\theta - S)^2 + \beta(Q_\theta - H)^2$$ Here, $S$ represents the standard one-step bootstrapped TD target used in DQN, while $H$ is the "best-ever" historical return retrieved from memory. By distilling these high-quality experiences directly into the network weights, the agent can adopt successful policies much faster than through standard TD propagation alone.
Experimental Results
The authors evaluated EMDQN on a benchmark suite of 57 Atari 2600 games within the ALE. Performance was compared against baselines including standard DQN and Double DQN (DDQN).
Key Findings
- Significant Sample Efficiency: EMDQN trained on only 40 million frames significantly outperformed standard DQN trained on 200 million frames, representing a roughly 5x speedup in learning.
- Superior Performance: Across the 57-game suite, EMDQN achieved a mean human-normalized score of 528.4% at 40M frames, whereas DQN reached only 151.2% at the same training stage.
- Stability and Regularization: The episodic memory target acts as a natural regularizer. In games like Asterix and Wizard, where standard DQN often suffers from value overestimation and performance collapse, EMDQN remains stable. The historical returns ($H$) provide a grounded "reality check" that prevents action-value estimates from inflating unrealistically.
Conclusion and Future Directions
EMDQN demonstrates that episodic memory can be effectively distilled into parametric models to accelerate RL. While highly effective in the near-deterministic environments of Atari, future work is needed to adapt the "best-ever" return strategy for highly stochastic environments where outliers might mislead the agent. Additionally, exploring dynamic weighting between the memory and inference targets could further optimize the balance between rapid learning and stable generalization.
Summary updated