Stabilizing Transformers for Reinforcement Learning

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, Raia Hadsell

Abstract

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP). However, the large-scale transformers used in NLP have yet to be successfully applied to the reinforcement learning (RL) setting. In this work, we demonstrate that the standard transformer architecture is difficult to optimize with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite.

The Challenge of Transformers in RL

In Reinforcement Learning, agents often operate in partially observable environments where they must maintain a memory of past observations to succeed. While the Long Short-Term Memory (LSTM) network has been the industry standard, Transformers theoretically offer a superior mechanism for jumping across long time horizons without vanishing gradients.

Despite this potential, standard Transformers often fail to learn effective policies in RL. The authors identify that the standard "Post-norm" architecture—where Layer Normalization occurs after the residual addition—is poorly suited for the high-variance, non-stationary nature of RL. At initialization, this creates a complex non-linear path that prevents the agent from learning basic "reactive" behaviors (like moving toward a visible goal) which are necessary precursors to more complex memory-based strategies.

Architectural Innovations: GTrXL

The researchers propose two primary modifications to the Transformer-XL architecture to create the GTrXL:

  1. Identity Map Reordering (TrXL-I): The authors move Layer Normalization to the input stream of the sub-modules (Pre-norm). This ensures that at the start of training, the sub-modules produce near-zero values, allowing the input to flow through the network essentially unchanged. This "identity map" allows the agent to start by learning simple Markovian (reactive) policies.
  2. Gating Mechanisms: Instead of a simple residual connection, they replace the addition with a GRU-type gating layer. This gives the network explicit control over how much new information to integrate versus how much of the previous state to preserve, further stabilizing gradient flow.
Comparison of the standard TrXL, the reordered TrXL-I, and the final GTrXL architecture which incorporates both identity map reordering and gating layers.

Key Results and Insights

The GTrXL was evaluated across several challenging benchmarks, most notably the DMLab-30 suite.

  • State-of-the-Art Performance: GTrXL achieved a mean human-normalized score of 117.6%, outperforming the competitive LSTM baseline (99.3%) and even surpassing MERLIN (115.2%), a specialized architecture with explicit external memory.
  • Learn to Walk Before You Remember: By favoring an identity map at initialization, the GTrXL acts like a shallow, reactive network early in training. As weights grow, it gradually "unlocks" its deep, multi-head attention capabilities.
  • Stability Across Task Types: GTrXL's stability and performance consistently match or exceed competitive baselines even on more reactive tasks where memory is less critical, proving it is a robust general-purpose architecture.
Mean human-normalized scores on DMLab30 (Reactive) tasks, showing that GTrXL maintains high stability and performance even in environments where memory is not the primary bottleneck.

Conclusion

GTrXL offers an easy-to-train, simple-to-implement, but substantially more expressive architectural alternative to the multi-layer LSTM. By addressing the optimization instabilities of standard self-attention, this work paves the way for applying large-scale Transformer models to complex, partially observable reinforcement learning domains.

Summary updated

Abstract

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical. GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially observable environments.
BibTeX
@inproceedings{parisotto2020stabilizing,
  author = {Emilio Parisotto and H. Francis Song and Jack W. Rae and Razvan Pascanu and Çaglar Gülçehre and Siddhant M. Jayakumar and Max Jaderberg and Raphael Lopez Kaufman and Aidan Clark and Seb Noury and Matthew M. Botvinick and Nicolas Heess and Raia Hadsell},
  title = {{Stabilizing Transformers for Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=px57hiuHms},
}
Figure