Transformer-based Working Memory for Multiagent Reinforcement Learning with Action Parsing

Yaodong Yang, Guangyong Chen, Weixun Wang, Xiaotian Hao, Jianye Hao, Pheng-Ann HengNeurIPS 2022 · OpenReview

Introduction

In Multiagent Reinforcement Learning (MARL), agents rarely have a full view of the world—a challenge known as partial observability. To make effective decisions, they must remember past events, such as the last known location of an enemy or the trajectory of a teammate.

The Agent Transformer Memory (ATM) network introduces a human-inspired "working memory" and a novel "action parsing" technique to MARL. By treating environmental entities and historical memory slots as a unified set of tokens, ATM overcomes the limitations of traditional Recurrent Neural Networks (RNNs), leading to state-of-the-art performance and faster convergence on complex benchmarks like StarCraft II.

The Problem with Traditional Memory

Traditionally, researchers have relied on Recurrent Neural Networks (RNNs), such as GRUs, to handle agent memory. However, RNNs suffer from a "bottleneck" problem: they compress the entire history into a single hidden vector. This compression loses the rich, "object-oriented" structure of multiagent environments, where the world is naturally composed of distinct entities like allies, enemies, and obstacles.

While Transformers are generally better at handling sequences and structured data, their computational cost grows quadratically as the history gets longer, making them difficult to deploy in long-running RL tasks.

The ATM Architecture

To address these challenges, the authors propose the Agent Transformer Memory (ATM), which consists of two primary innovations:

ATM Architecture

1. Transformer-based Working Memory

Instead of processing an ever-growing history, ATM maintains a fixed-capacity memory buffer (slots). Inspired by human working memory, it treats these memory slots and current environmental entities (self, allies, enemies) as a unified set of input tokens for a Transformer block.

At each step, the oldest memory slot is dropped, and a new one—generated from the Transformer’s output—is added. This working memory updating schema keeps computational costs constant while allowing the agent to reason across both space (entities) and time (memory). Interestingly, the researchers found that a very small working memory (just 3 slots) is often enough to significantly outperform RNNs, suggesting that "recent" high-quality memory is more valuable than "long" noisy memory.

2. Entity-Bound Action Layer (Action Parsing)

Standard RL models usually map a state vector to a list of action probabilities. ATM introduces a "semantic inductive bias" by binding actions to specific entities. For example, the "Attack Enemy 1" action is explicitly linked to the embedding of "Enemy 1." This allows the network to share logic across similar actions and makes the agent's decision-making process more explainable. By looking at the Transformer's attention maps, we can see exactly which entities or past memories influenced a specific command.

Key Results

The researchers evaluated ATM on two popular and challenging MARL benchmarks: the StarCraft Multi-Agent Challenge (SMAC) and Level-Based Foraging (LBF). ATM acts as a "plug-and-play" architecture that can be integrated into existing algorithms (like QMIX, QPLEX, or MAPPO) to boost their performance.

SMAC Benchmark Results
  • Breakthroughs on "Super Hard" Maps: On the SMAC corridor and 6h_vs_8z maps—where standard GRU-based algorithms like QMIX and QPLEX often fail to learn at all—ATM achieved near-100% win rates.
  • Learning Acceleration: In Level-Based Foraging, ATM-equipped agents reached peak performance significantly faster than those using MLPs or GRUs.

Limitations and Future Work

While ATM shows impressive results, it does have limitations. Currently, the "Entity-Bound Action Layer" requires researchers to manually define which actions belong to which entities. Future work could involve learning these mappings automatically.

Additionally, while the fixed-capacity memory helps with efficiency, extremely large-scale environments with hundreds of entities would still challenge the Transformer's self-attention mechanism. Finally, the authors propose that the semantic richness of this architecture could eventually allow for the integration of advanced Natural Language Processing (NLP) techniques, helping agents understand complex environmental rules or instructions.

Summary updated

Abstract

Learning in real-world multiagent tasks is challenging due to the usual partial observability of each agent. Previous efforts alleviate the partial observability by historical hidden states with Recurrent Neural Networks, however, they do not consider the multiagent characters that either the multiagent observation consists of a number of object entities or the action space shows clear entity interactions. To tackle these issues, we propose the Agent Transformer Memory (ATM) network with a transformer-based memory. First, ATM utilizes the transformer to enable the unified processing of the factored environmental entities and memory. Inspired by the human’s working memory process where a limited capacity of information temporarily held in mind can effectively guide the decision-making, ATM updates its fixed-capacity memory with the working memory updating schema. Second, as agents' each action has its particular interaction entities in the environment, ATM parses the action space to introduce this action’s semantic inductive bias by binding each action with its specified involving entity to predict the state-action value or logit. Extensive experiments on the challenging SMAC and Level-Based Foraging environments validate that ATM could boost existing multiagent RL algorithms with impressive learning acceleration and performance improvement.
BibTeX
@inproceedings{yang2022transformer,
  author = {Yaodong Yang and Guangyong Chen and Weixun Wang and Xiaotian Hao and Jianye HAO and Pheng-Ann Heng},
  title = {{Transformer-based Working Memory for Multiagent Reinforcement Learning with Action Parsing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=pd6ipu3jDw},
}
Figure