Introduction
In Multiagent Reinforcement Learning (MARL), agents rarely have a full view of the world—a challenge known as partial observability. To make effective decisions, they must remember past events, such as the last known location of an enemy or the trajectory of a teammate.
The Agent Transformer Memory (ATM) network introduces a human-inspired "working memory" and a novel "action parsing" technique to MARL. By treating environmental entities and historical memory slots as a unified set of tokens, ATM overcomes the limitations of traditional Recurrent Neural Networks (RNNs), leading to state-of-the-art performance and faster convergence on complex benchmarks like StarCraft II.
The Problem with Traditional Memory
Traditionally, researchers have relied on Recurrent Neural Networks (RNNs), such as GRUs, to handle agent memory. However, RNNs suffer from a "bottleneck" problem: they compress the entire history into a single hidden vector. This compression loses the rich, "object-oriented" structure of multiagent environments, where the world is naturally composed of distinct entities like allies, enemies, and obstacles.
While Transformers are generally better at handling sequences and structured data, their computational cost grows quadratically as the history gets longer, making them difficult to deploy in long-running RL tasks.
The ATM Architecture
To address these challenges, the authors propose the Agent Transformer Memory (ATM), which consists of two primary innovations:
1. Transformer-based Working Memory
Instead of processing an ever-growing history, ATM maintains a fixed-capacity memory buffer (slots). Inspired by human working memory, it treats these memory slots and current environmental entities (self, allies, enemies) as a unified set of input tokens for a Transformer block.
At each step, the oldest memory slot is dropped, and a new one—generated from the Transformer’s output—is added. This working memory updating schema keeps computational costs constant while allowing the agent to reason across both space (entities) and time (memory). Interestingly, the researchers found that a very small working memory (just 3 slots) is often enough to significantly outperform RNNs, suggesting that "recent" high-quality memory is more valuable than "long" noisy memory.
2. Entity-Bound Action Layer (Action Parsing)
Standard RL models usually map a state vector to a list of action probabilities. ATM introduces a "semantic inductive bias" by binding actions to specific entities. For example, the "Attack Enemy 1" action is explicitly linked to the embedding of "Enemy 1." This allows the network to share logic across similar actions and makes the agent's decision-making process more explainable. By looking at the Transformer's attention maps, we can see exactly which entities or past memories influenced a specific command.
Key Results
The researchers evaluated ATM on two popular and challenging MARL benchmarks: the StarCraft Multi-Agent Challenge (SMAC) and Level-Based Foraging (LBF). ATM acts as a "plug-and-play" architecture that can be integrated into existing algorithms (like QMIX, QPLEX, or MAPPO) to boost their performance.
- Breakthroughs on "Super Hard" Maps: On the SMAC corridor and 6h_vs_8z maps—where standard GRU-based algorithms like QMIX and QPLEX often fail to learn at all—ATM achieved near-100% win rates.
- Learning Acceleration: In Level-Based Foraging, ATM-equipped agents reached peak performance significantly faster than those using MLPs or GRUs.
Limitations and Future Work
While ATM shows impressive results, it does have limitations. Currently, the "Entity-Bound Action Layer" requires researchers to manually define which actions belong to which entities. Future work could involve learning these mappings automatically.
Additionally, while the fixed-capacity memory helps with efficiency, extremely large-scale environments with hundreds of entities would still challenge the Transformer's self-attention mechanism. Finally, the authors propose that the semantic richness of this architecture could eventually allow for the integration of advanced Natural Language Processing (NLP) techniques, helping agents understand complex environmental rules or instructions.
Summary updated