Think Before You Act: Decision Transformers with Internal Working Memory

Jikun Kang, Romain Laroche, Xindi Yuan, Adam Trischler, Xue Liu, Jie Fu

Introduction

Decision Transformers (DT) have emerged as a powerful paradigm for Reinforcement Learning (RL) by framing the problem as sequence modeling. However, standard DT architectures suffer from a significant limitation: they store task-specific knowledge implicitly within their neural network weights. This reliance on implicit memory leads to "catastrophic forgetting" when models are trained on multiple tasks and necessitates massive parameter scaling to achieve generalization.

This paper introduces DT-Mem, an architecture that equips Decision Transformers with an explicit, content-addressable working memory module. By decoupling "how to act" (stored in weights) from "what is happening" (stored in memory), DT-Mem achieves superior generalization and significant improvements in training efficiency.

Methodology: The DT-Mem Architecture

The core innovation of DT-Mem is a memory module inspired by Neural Turing Machines (NTMs) and cognitive psychology. Unlike traditional transformers that process sequences in a purely feed-forward manner, DT-Mem maintains a persistent internal state.

Detailed architecture of the DT-Mem memory module, illustrating the update and retrieval mechanisms.

The architecture consists of several key components:

  1. Working Memory Matrix ($M$): A dedicated matrix where each row serves as a "memory slot" for storing task-specific information.
  2. Content-Based Addressing: An attention mechanism compares the current input sequence against the memory matrix to identify and retrieve relevant information.
  3. Memory Update (Erase & Add): To integrate new experiences, the model computes an "erasing vector" to clear outdated information and an "adding vector" to write new data. This allows the agent to "blend" current observations with past knowledge dynamically.
  4. Parameter-Efficient Adaptation: For new tasks, the model utilizes Low-Rank Adaptation (LoRA) specifically on the memory module. This allows the agent to adapt by updating only a tiny fraction (approximately 0.7%) of its parameters, preserving the foundational knowledge of the pre-trained model.

Experimental Results and Scaling Efficiency

The authors evaluated DT-Mem on the Atari benchmark (including held-out games like Alien and Ms. Pac-Man) and the Meta-World (ML45) robotic manipulation suite.

Comparison of Human-Normalized IQM Scores against model parameters, highlighting the scaling efficiency of DT-Mem.

The results demonstrate a stark contrast in efficiency compared to standard Multi-game Decision Transformers (MDT):

  • Generalization: On held-out Atari games, DT-Mem achieved a DQN-normalized score of 51.0% on Alien, vastly outperforming the 3.8% achieved by the MDT baseline.
  • Scaling Efficiency: As shown in the performance analysis, a 20M parameter DT-Mem performs on par with a 200M parameter MDT. This suggests that explicit memory is a far more efficient scaling path for agent intelligence than simple parameter bloat.
  • Training Speed: DT-Mem (20M) reached comparable performance levels in just 50 hours of training, a 32x speedup over the 1600 hours required by the MDT-200M baseline.

Key Insights and Future Directions

The "Think Before You Act" paradigm enabled by the memory module allows agents to reason over past trajectories before committing to an action. This leads to more robust strategies, particularly in game sequels or variations. Furthermore, DT-Mem exhibits higher resilience to input noise, as the memory module helps stabilize decision-making by recalling consistent past patterns.

Despite these successes, challenges remain. Current fine-tuning still requires roughly 10% of a new task's dataset, and the model is currently limited to offline RL settings. Future work will focus on improving sample efficiency and extending the architecture to online exploration, where the agent must actively collect its own data in novel environments.

Summary updated

Abstract

Decision Transformer-based decision-making agents have shown the ability to generalize across multiple tasks. However, their performance relies on massive data and computation. We argue that this inefficiency stems from the forgetting phenomenon, in which a model memorizes its behaviors in parameters throughout training. As a result, training on a new task may deteriorate the model's performance on previous tasks. In contrast to LLMs' implicit memory mechanism, the human brain utilizes distributed memory storage, which helps manage and organize multiple skills efficiently, mitigating the forgetting phenomenon. Inspired by this, we propose a working memory module to store, blend, and retrieve information for different downstream tasks. Evaluation results show that the proposed method improves training efficiency and generalization in Atari games and Meta-World object manipulation tasks. Moreover, we demonstrate that memory fine-tuning further enhances the adaptability of the proposed architecture.
BibTeX
@inproceedings{kang2024think,
  author = {Jikun Kang and Romain Laroche and Xindi Yuan and Adam Trischler and Xue Liu and Jie Fu},
  title = {{Think Before You Act: Decision Transformers with Internal Working Memory}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=PSQ5Z920M8},
}
Figure