Stabilizing Transformers for Reinforcement Learning

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, Raia Hadsell

Introduction

In Reinforcement Learning (RL), agents frequently operate in partially observable environments where they must remember past observations to make optimal decisions. For years, the Long Short-Term Memory (LSTM) network has been the standard architecture for providing this memory.

Transformers offer a compelling alternative due to their ability to model much longer temporal horizons than LSTMs, a feature that has revolutionized Natural Language Processing (NLP). However, the standard Transformer architecture is notoriously fragile when trained with RL objectives. RL training is often high-variance and non-stationary. Consequently, the standard Transformer's reliance on specific learning rate warmups and sensitive residual connections frequently causes the model to fail, often learning nothing better than a random policy.

To address this, researchers introduce the Gated Transformer-XL (GTrXL), which utilizes specific architectural modifications to provide the stability necessary for RL, allowing Transformers to finally outperform LSTMs and specialized memory architectures on complex benchmarks.

Methodology: Architectural Modifications

The authors propose two primary architectural shifts to stabilize the model, evolving the standard Transformer-XL (TrXL) into the Gated Transformer-XL (GTrXL):

  1. Identity Map Reordering (TrXL-I): In a standard Transformer, Layer Normalization (LN) is applied after the residual connection. The authors move LN to the input stream of the submodules (Multi-Head Attention and MLP). This ensures that at initialization, there is a direct, un-transformed "identity" path from the input of the first layer to the output of the last.
  2. Gating Layers: The standard residual connection ($y = x + f(x)$) is replaced with a gating mechanism. After experimenting with several types, the authors found a GRU-type gating unit to be the most successful. This allows the network to explicitly control how much information from the current layer's computation is merged into the main hidden state.
The evolution of the architecture from the canonical Transformer (left) to the GTrXL (right).

Experimental Setup

The researchers evaluated the GTrXL architecture across several challenging benchmarks:

  • DMLab-30: A large-scale suite of 30 first-person 3D environments involving navigation, logic, and memory.
  • Numpad: A continuous control task designed to test how well an agent can scale its memory as the "horizon" (the number of items to remember) increases combinatorially.
  • Memory Maze: A navigation task where the agent must find and remember the location of an object in a maze, even after being teleported.

Key Findings and Insights

The "Walk Before You Remember" Hypothesis

The authors argue that in RL, an agent needs to learn simple "Markovian" behaviors (like walking forward when it sees a hallway) before it can learn to effectively use memory. By reordering the LayerNorm, the network starts as an identity map, effectively presenting the agent with a simple, easy-to-learn policy at the start of training. Standard Transformers, by contrast, transform the state immediately, forcing the agent to deal with complex representations before it has even learned basic movement.

Gating as a Stabilizer

In the noisy optimization landscape of RL, a simple addition (like a standard residual connection) can be highly disruptive. Gating acts as a "damper," allowing the network to slowly integrate attention-based information only when it is strictly useful for reducing the RL loss. This makes GTrXL far less sensitive to hyperparameters and random seeds, positioning it as a viable "drop-in" replacement for LSTMs.

State-of-the-Art Performance

GTrXL achieved a human-normalized score of 117.6% on DMLab-30, significantly outperforming the LSTM baseline (99.3%). It even exceeded MERLIN (115.2%), a highly specialized architecture that relies on an external memory bank and auxiliary losses. Furthermore, on the Numpad task, GTrXL maintained high performance as memory requirements grew, whereas the LSTM's performance plummeted.

Learning curves on DMLab-30 showing GTrXL (orange) significantly outperforming LSTMs (blue) and the failing standard TrXL (green).

Limitations and Future Work

While GTrXL is more sample-efficient—learning more from fewer environment steps—Transformers generally incur a higher computational cost per training step than LSTMs. Additionally, this work focused on 12-layer models; future research is needed to determine if these stabilization techniques hold up when scaling to the massive, multi-billion parameter sizes seen in modern Large Language Models. Finally, while GTrXL excels at memory-heavy tasks, its performance gains are more modest in purely "reactive" tasks where memory isn't strictly required, though it still matches or slightly exceeds LSTM baselines in those scenarios.

Summary updated

Abstract

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP). Harnessing the transformer’s ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical.
BibTeX
@inproceedings{parisotto2020stabilizing,
  author = {Emilio Parisotto and H. Francis Song and Jack W. Rae and Razvan Pascanu and Çaglar Gülçehre and Siddhant M. Jayakumar and Max Jaderberg and Raphaël Lopez Kaufman and Aidan Clark and Seb Noury and Matthew M. Botvinick and Nicolas Heess and Raia Hadsell},
  title = {{Stabilizing Transformers for Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=px57hiuHms},
}
Figure