Transfer Reinforcement Learning Using Output-Gated Working Memory

Arthur Williams, Joshua L. PhillipsAAAI 2020 · DOI

Introduction

Reinforcement learning (RL) agents often struggle with "brittleness," requiring extensive retraining for new tasks even when they share significant logic with previously learned ones. In contrast, biological systems excel at transfer learning by abstracting general rules. This research, presented at AAAI-20, explores how the biological mechanism of "output gating" in working memory (WM) can facilitate this process, allowing agents to form shared representations that apply across diverse scenarios.

Methods: Dual-Gate Architecture

The authors extended the Holographic Working Memory Toolkit (HWMtk), utilizing Holographic Reduced Representations (HRRs) to bind features like color and location into high-dimensional vectors. The core innovation is a dual-gate system that mimics prefrontal cortex functions:

  1. Input Gate: Determines which perceived stimuli (e.g., a color cue) should be stored in working memory.
  2. Output Gate: Controls when that stored information is released to the action-selection layer to influence behavior.
Schematic of the output-gated working memory architecture showing the flow from cue perception to action selection via input and output gates.

A key constraint is "memory flushing": once the output gate opens and the memory is utilized, the slot is cleared. This forces the agent to learn the specific context in which a memory is useful, preventing over-reliance on task-specific cues during general navigation phases. The model uses Temporal Difference (TD) learning to adjust the weights of these gates based on rewards.

Experimental Setup

The model was evaluated using a 5x5 grid-world maze divided by a barrier with a single central opening. The agent starts in the bottom half and receives a color cue (Red, Green, or Purple) indicating the goal location in the top half.

The Red, Green, and Purple grid-world tasks. All tasks share a common bottleneck (the opening in the barrier) but have different goal states.
  • Red Task: Goal in the top-left.
  • Green Task: Goal in the top-right.
  • Purple Task: Goal in the top-middle.

The transfer challenge involves training the agent on "source" tasks (Red and Green) and measuring how quickly it masters a "target" task (Purple). Because all tasks require navigating to the bottleneck first, the agent can potentially transfer the "navigation to the door" skill.

Results and Key Insights

The output-gated model significantly outperformed standard input-gated architectures in transfer efficiency.

  • One-Shot Transfer: Most notably, the output-gated model achieved the performance threshold for the Purple task in just one episode after learning the Red and Green tasks. In comparison, a baseline model without transfer required approximately 32,000 episodes, and an input-gate-only transfer model required 2,750 episodes.
  • Shared Representations: The output gate learned to remain closed in the bottom half of the maze, where the cue was irrelevant for reaching the bottleneck. This allowed the agent to treat initial navigation as a universal skill, separate from the task-specific navigation required after passing the barrier.
  • Jumpstart Performance: The model showed nearly double the "jumpstart" improvement (initial performance on a new task) compared to input-gated models.

Conclusion

By incorporating output gating, RL agents can develop a form of episodic memory that understands the context of its own decisions. While currently tested in a simplified grid-world, this approach demonstrates a path toward more robust, human-like transfer learning. Future work will focus on scaling this logic to high-dimensional sensory inputs and validating it against complex cognitive benchmarks like the AX-CPT task.

Summary updated

BibTeX
@inproceedings{williams2020transfer,
  author = {Arthur Williams and Joshua L. Phillips},
  title = {{Transfer Reinforcement Learning Using Output-Gated Working Memory}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i02.5488},
}
Figure