Introduction
In Reinforcement Learning (RL), agents frequently operate in partially observable environments where they must remember past observations to make optimal decisions. For years, the Long Short-Term Memory (LSTM) network has been the standard architecture for providing this memory.
Transformers offer a compelling alternative due to their ability to model much longer temporal horizons than LSTMs, a feature that has revolutionized Natural Language Processing (NLP). However, the standard Transformer architecture is notoriously fragile when trained with RL objectives. RL training is often high-variance and non-stationary. Consequently, the standard Transformer's reliance on specific learning rate warmups and sensitive residual connections frequently causes the model to fail, often learning nothing better than a random policy.
To address this, researchers introduce the Gated Transformer-XL (GTrXL), which utilizes specific architectural modifications to provide the stability necessary for RL, allowing Transformers to finally outperform LSTMs and specialized memory architectures on complex benchmarks.
Methodology: Architectural Modifications
The authors propose two primary architectural shifts to stabilize the model, evolving the standard Transformer-XL (TrXL) into the Gated Transformer-XL (GTrXL):
- Identity Map Reordering (TrXL-I): In a standard Transformer, Layer Normalization (LN) is applied after the residual connection. The authors move LN to the input stream of the submodules (Multi-Head Attention and MLP). This ensures that at initialization, there is a direct, un-transformed "identity" path from the input of the first layer to the output of the last.
- Gating Layers: The standard residual connection ($y = x + f(x)$) is replaced with a gating mechanism. After experimenting with several types, the authors found a GRU-type gating unit to be the most successful. This allows the network to explicitly control how much information from the current layer's computation is merged into the main hidden state.
Experimental Setup
The researchers evaluated the GTrXL architecture across several challenging benchmarks:
- DMLab-30: A large-scale suite of 30 first-person 3D environments involving navigation, logic, and memory.
- Numpad: A continuous control task designed to test how well an agent can scale its memory as the "horizon" (the number of items to remember) increases combinatorially.
- Memory Maze: A navigation task where the agent must find and remember the location of an object in a maze, even after being teleported.
Key Findings and Insights
The "Walk Before You Remember" Hypothesis
The authors argue that in RL, an agent needs to learn simple "Markovian" behaviors (like walking forward when it sees a hallway) before it can learn to effectively use memory. By reordering the LayerNorm, the network starts as an identity map, effectively presenting the agent with a simple, easy-to-learn policy at the start of training. Standard Transformers, by contrast, transform the state immediately, forcing the agent to deal with complex representations before it has even learned basic movement.
Gating as a Stabilizer
In the noisy optimization landscape of RL, a simple addition (like a standard residual connection) can be highly disruptive. Gating acts as a "damper," allowing the network to slowly integrate attention-based information only when it is strictly useful for reducing the RL loss. This makes GTrXL far less sensitive to hyperparameters and random seeds, positioning it as a viable "drop-in" replacement for LSTMs.
State-of-the-Art Performance
GTrXL achieved a human-normalized score of 117.6% on DMLab-30, significantly outperforming the LSTM baseline (99.3%). It even exceeded MERLIN (115.2%), a highly specialized architecture that relies on an external memory bank and auxiliary losses. Furthermore, on the Numpad task, GTrXL maintained high performance as memory requirements grew, whereas the LSTM's performance plummeted.
Limitations and Future Work
While GTrXL is more sample-efficient—learning more from fewer environment steps—Transformers generally incur a higher computational cost per training step than LSTMs. Additionally, this work focused on 12-layer models; future research is needed to determine if these stabilization techniques hold up when scaling to the massive, multi-billion parameter sizes seen in modern Large Language Models. Finally, while GTrXL excels at memory-heavy tasks, its performance gains are more modest in purely "reactive" tasks where memory isn't strictly required, though it still matches or slightly exceeds LSTM baselines in those scenarios.
Summary updated