Episodic Memory Deep Q-Networks

Zichuan Lin, Tianqi Zhao, Guangwen Yang, Lintao Zhang

Introduction

Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains like the Arcade Learning Environment (ALE). However, a persistent challenge remains: sample inefficiency. Standard algorithms such as Deep Q-Networks (DQN) often require hundreds of millions of environment interactions to converge. This inefficiency stems from the reliance on "slow" gradient-based optimization and one-step temporal difference (TD) learning, where rewards propagate backward through state-action chains incrementally.

In contrast, biological systems utilize episodic memory—often associated with the hippocampus—to rapidly encode and retrieve successful experiences. The Episodic Memory Deep Q-Network (EMDQN) bridges this gap by integrating a biologically-inspired episodic memory module with the generalization capabilities of deep neural networks.

Methodology: Integrating Memory and Generalization

EMDQN maintains a dual-system architecture: a parametric Q-network for generalization and a non-parametric memory table for rapid learning from high-reward experiences.

Architecture of EMDQN showing the parallel paths of standard Q-learning and episodic memory lookup via random projection.

1. Architecture and Fast Lookup

The system utilizes a standard Convolutional Neural Network (CNN) to predict action values ($Q_\theta$). Simultaneously, it maintains an Episodic Memory Table ($H$) that stores the maximum Monte-Carlo return—the total actual reward from a state to the episode's end—ever encountered for specific state-action pairs. To ensure computational efficiency, high-dimensional state images are mapped to a low-dimensional space using a fixed Gaussian random projection, enabling rapid exact-match lookups without the overhead of K-Nearest Neighbors searches.

2. Dual-Target Loss Function

The primary innovation of EMDQN is its loss function, which supervises the Q-network using two distinct targets: $$L = \alpha(Q_\theta - S)^2 + \beta(Q_\theta - H)^2$$ Here, $S$ represents the standard one-step bootstrapped TD target used in DQN, while $H$ is the "best-ever" historical return retrieved from memory. By distilling these high-quality experiences directly into the network weights, the agent can adopt successful policies much faster than through standard TD propagation alone.

Experimental Results

The authors evaluated EMDQN on a benchmark suite of 57 Atari 2600 games within the ALE. Performance was compared against baselines including standard DQN and Double DQN (DDQN).

Comparison of training scores between standard DQN and EMDQN across six Atari games, demonstrating significant sample efficiency gains.

Key Findings

  • Significant Sample Efficiency: EMDQN trained on only 40 million frames significantly outperformed standard DQN trained on 200 million frames, representing a roughly 5x speedup in learning.
  • Superior Performance: Across the 57-game suite, EMDQN achieved a mean human-normalized score of 528.4% at 40M frames, whereas DQN reached only 151.2% at the same training stage.
  • Stability and Regularization: The episodic memory target acts as a natural regularizer. In games like Asterix and Wizard, where standard DQN often suffers from value overestimation and performance collapse, EMDQN remains stable. The historical returns ($H$) provide a grounded "reality check" that prevents action-value estimates from inflating unrealistically.

Conclusion and Future Directions

EMDQN demonstrates that episodic memory can be effectively distilled into parametric models to accelerate RL. While highly effective in the near-deterministic environments of Atari, future work is needed to adapt the "best-ever" return strategy for highly stochastic environments where outliers might mislead the agent. Additionally, exploring dynamic weighting between the memory and inference targets could further optimize the balance between rapid learning and stable generalization.

Summary updated

Abstract

Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently, episodic memory based RL has attracted attention due to its ability to latch on good actions quickly. In this paper, we present a simple yet effective biologically inspired RL algorithm called Episodic Memory Deep Q-Networks (EMDQN), which leverages episodic memory to supervise an agent during training. Experiments show that our proposed method can lead to better sample efficiency and is more likely to find good policies. It only requires 1/5 of the interactions of DQN to achieve many state-of-the-art performances on Atari games, significantly outperforming regular DQN and other episodic memory based RL algorithms.
BibTeX
@inproceedings{lin2018episodic,
  author = {Zichuan Lin and Tianqi Zhao and Guangwen Yang and Lintao Zhang},
  title = {{Episodic Memory Deep Q-Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/337},
}
Figure