Episodic Control as Meta-Reinforcement Learning

Samuel B. Ritter, J. S. Wang, Zeb Kurth‐Nelson, Matthew BotvinickbioRxiv 2018 · DOI

TL;DR

This paper introduces Episodic Meta-Reinforcement Learning (EMRL), a novel framework that integrates episodic memory into a meta-learning agent. EMRL learns to dynamically deploy both incremental and episodic model-based learning strategies, mirroring human decision-making and achieving better performance on tasks requiring memory of past experiences.

What Problem Does This Paper Address?

Human learning is remarkably flexible, drawing on multiple strategies. While reinforcement learning (RL) research has traditionally focused on model-free (learning from trial-and-error) and model-based (learning a world model) approaches, recent work suggests episodic memory—recalling specific past experiences—also plays a crucial role. The challenge is to create a unified computational framework that explains how these different learning strategies are coordinated and deployed by a single learning system, especially in novel situations. Existing meta-RL approaches, while capable of learning to learn, haven't fully incorporated the distinct contributions of episodic memory.

Methods

The authors extend the meta-reinforcement learning (meta-RL) framework by equipping a recurrent neural network (RNN), specifically a Long Short-Term Memory (LSTM) network, with an episodic memory system. This system is inspired by theories of memory reinstatement and gating.

Here's a breakdown of the key components:

  • Working Memory (LSTM): The core of the agent, incrementally updating its internal state based on environmental inputs, actions, and rewards.
  • Long-Term Memory (LTM) Storage: At the end of each trial, the agent stores its current working memory state along with a perceptual representation of the context.
  • LTM Retrieval: On each time step, the agent searches its LTM for contexts similar to the current one and retrieves the associated working memory activations.
  • Gating Mechanism: A learned "r-gate" controls how much of the retrieved LTM information is integrated into the current working memory. This gate is crucial for learning when to use episodic memory and when to rely on incremental learning.

The entire system is trained using a model-free meta-learning objective across a distribution of tasks, allowing it to learn how to deploy and coordinate these different learning strategies.

Datasets

The paper uses a modified version of the "two-step task" (originally from Daw et al., 2011) augmented with episodic cues. This task is designed to specifically dissociate the effects of incremental and episodic learning. The training involved 10,000 episodes of 100 trials each, and evaluation was performed on 500 additional episodes with fixed weights.

Key Insights and Ideas

  • Episodic Memory as a Learned Strategy: The core insight is that episodic memory isn't just a passive storage system but an active learning strategy that can be learned and deployed by a meta-learning agent. The learned gating mechanism is key to this.
  • Unified Learning Framework: EMRL provides a compelling computational account of how incremental (model-free and model-based) and episodic (model-free and model-based) learning can emerge from a single, unified meta-learning architecture.
  • Model-Based Advantage: The model demonstrates that for tasks with episodic structure, model-based strategies (both incremental and episodic) are more beneficial than model-free ones, aligning with observed human behavior.
  • Behavioral Parallels: The EMRL agent's behavior on the two-step task closely mirrors human performance as reported in prior studies, suggesting EMRL is a plausible model for human decision-making.

Key Results

  • Improved Performance on Cued Trials: EMRL significantly outperformed a control meta-RL agent (MRL) that lacked episodic memory, particularly on cued trials where past experiences were relevant. This highlights EMRL's ability to leverage episodic information.
  • Canonical Learning Patterns: EMRL exhibited both incremental model-based learning (indicated by the interaction between reward and transition type) and episodic model-based learning on the cued trials.
  • Absence of Episodic Model-Free Learning: Consistent with human data, EMRL did not show significant evidence of episodic model-free learning.
  • Learned Gating: Analysis of the r-gate activations showed that episodic memory retrieval was more strongly utilized during cued trials and, specifically, during the first stage of the trial where action selection occurs.

Limitations and Future Work

  • Small Gating Magnitudes: While the r-gates learned to modulate episodic memory use, their overall magnitude was small. The authors suggest that other mechanisms might be at play, or that regularization techniques (like dropout) could lead to more interpretable gating.
  • Generalizability of Training Distribution: The current success is tied to the specific episodic two-step task. Future work could explore more ecologically inspired training distributions to see if the model can reproduce a wider range of human learning behaviors and deviations from optimality.
  • Biological Plausibility: The authors suggest further empirical work to test predictions about pattern reinstatement during memory retrieval of sequences and computations, and to investigate the biological mechanisms of gating reinstated information. They also propose exploring how training participants on tasks where episodic memory is either useful or distracting might modulate pattern reinstatement in the brain.

Summary updated

Abstract

Abstract Recent research has placed episodic reinforcement learning (RL) alongside model-free and model-based RL on the list of processes centrally involved in human reward-based learning. In the present work, we extend the unified account of model-free and model-based RL developed by Wang et al. (2018) to further integrate episodic learning. In this account, a generic model-free “meta-learner” learns to deploy and coordinate among all of these learning algorithms. The meta-learner learns through brief encounters with many novel tasks, so that it learns to learn about new tasks. We show that when equipped with an episodic memory system inspired by theories of reinstatement and gating, the meta-learner learns to use the episodic and model-based learning algorithms observed in humans in a task designed to dissociate among the influences of various learning strategies. We discuss implications and predictions of the model.
BibTeX
@misc{ritter2018episodic,
  author = {Samuel B. Ritter and J. S. Wang and Zeb Kurth‐Nelson and Matthew Botvinick},
  title = {{Episodic Control as Meta-Reinforcement Learning}},
  year = {2018},
  doi = {10.1101/360537},
  howpublished = {bioRxiv},
}

Loading related content...

Figure