Abstract
Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently, episodic memory based RL has attracted attention due to its ability to latch on good actions quickly. In this paper, we present a simple yet effective biologically inspired RL algorithm called Episodic Memory Deep Q-Networks (EMDQN), which leverages episodic memory to supervise an agent during training. Experiments show that our proposed method can lead to better sample efficiency and is more likely to find good policies. It only requires 1/5 of the interactions of DQN to achieve many state-of-the-art performances on Atari games, significantly outperforming regular DQN and other episodic memory based RL algorithms.
Introduction
Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains such as Go and video games. However, these methods are notoriously sample inefficient, often requiring hundreds of millions of interactions to learn effective policies. This inefficiency stems largely from "bootstrapping"—the practice of updating current value estimates based on other estimates of future rewards—which slows down reward propagation. While episodic control (EC) methods can quickly memorize high-reward sequences, they typically lack the generalization capabilities of neural networks.
Methods: The EMDQN Architecture
The Episodic Memory Deep Q-Network (EMDQN) bridges the gap between parametric neural networks and non-parametric memory tables. Inspired by the dual-learning systems of the human brain—the striatum (habits) and the hippocampus (episodic memory)—EMDQN uses memory to supervise the training of a deep Q-network.
Dual-Target Loss Function
EMDQN optimizes a loss function that combines two distinct targets:
- Inference Target ($S$): The standard one-step bootstrapped target used in traditional DQN.
- Memory Target ($H$): The maximum Monte-Carlo return ever recorded for a specific state-action pair.
Memory and Distillation
The system maintains a memory table that stores the best historical returns. To ensure efficiency, high-dimensional game states are compressed into small vectors using Random Projection. The episodic memory acts as a supervisor, allowing the neural network to "distill" knowledge from the best-ever experiences into its weights, facilitating faster learning and better generalization.
Key Insights
- Accelerated Reward Propagation: By utilizing full Monte-Carlo returns from the memory table, rewards propagate through the network significantly faster than the incremental approach of standard DQN.
- Mitigating Overestimation: Standard Q-learning often suffers from over-optimistic value estimates. The memory table provides a "reality check" by recording actual historical returns, which helps regularize the network and prevent divergence.
- Decoupled Learning Signal: EMDQN targets the best historical performance independently of the current network's fluctuations, providing a stable and concentrated learning signal.
Experimental Results
The algorithm was evaluated on the Arcade Learning Environment (ALE) across 57 Atari 2600 games. EMDQN was compared against baselines including DQN, Double-DQN (DDQN), and episodic methods like Neural Episodic Control (NEC).
Performance and Efficiency
- 5x Sample Efficiency: EMDQN achieved superior performance using only 40 million frames compared to the 200 million frames required by standard DQN.
- Benchmark Scores: At the 40M frame mark, EMDQN reached a mean human-normalized score of 528.4%, significantly higher than DQN (151.2%) and NEC (144.8%).
- Stability: EMDQN demonstrated remarkable stability in games like Asterix and Wizard of Wor, where standard DQN often experiences performance crashes due to "Q-value blow-up."
Conclusion
EMDQN offers a simple yet powerful modification to the standard deep RL framework. By integrating episodic memory as a supervisory signal, it achieves state-of-the-art results with a fraction of the data. Future work may explore dynamic tuning of the memory-inference weight and the application of EMDQN to highly stochastic environments.
Summary updated