Do Transformer World Models Give Better Policy Gradients?

Michel Ma, Tianwei Ni, Clement Gehring, Pierluca D'Oro, Pierre-Luc Bacon

Model-based reinforcement learning (RL) often relies on training a "world model" to mimic the environment. To improve a control policy, this model is unrolled into the future, and the reward signal is backpropagated through the model's predictions to update the policy's parameters. However, this approach frequently breaks down over long horizons. In complex or chaotic systems, the optimization landscape becomes highly irregular, causing gradients to either vanish to zero or explode to infinity.

Given that Transformers have revolutionized long-range dependency handling in sequence modeling, they seem like a natural solution. Yet, simply plugging Transformers in as world models hasn't yielded the expected breakthrough in RL. A recent paper presented at ICML 2024 investigates why this happens and proposes a structural fix to unlock the true potential of Transformers in long-horizon RL.

The Flaw in Standard World Models

The authors identify a fundamental structural flaw in how world models are typically deployed. Most standard models are History-conditioned World Models (HWMs) or Markovian models. To predict the next state, they autoregressively take the previously predicted state as an input.

While this makes intuitive sense for forward prediction, it creates a massive problem for backpropagation. When calculating policy gradients, this autoregressive loop forces the gradients to travel through "circuitous paths," compounding errors and gradient instabilities exponentially over time. Even a perfect Transformer becomes effectively recurrent—and thus unstable—if its output state is continuously fed back in as an input.

Comparison of gradient paths in Markovian, History-conditioned, and Action-conditioned world models. Red arrows highlight the circuitous gradient paths that cause instability in standard models.

The Solution: Action-Conditioned World Models (AWMs)

To resolve this, the researchers propose switching to Action-conditioned World Models (AWMs).

Instead of predicting the next state based on the immediate prior state ($\hat{s}_{t+1} = f(\hat{s}_t, a_t)$), an AWM predicts future states using only the initial state and the sequence of actions taken so far: $\hat{s}_{t+1} = g(s_1, a_1, \dots, a_t)$.

By conditioning only on the initial state and the action sequence, the gradient flows directly from the reward through the Transformer's attention mechanism straight to the actions. The authors prove theoretically that while standard models suffer from exponential gradient growth ($O(L_s^H)$), a Transformer-based AWM exhibits a much more manageable polynomial growth ($O(H^3)$).

Key Insights and Performance

Surprisingly, AWMs can produce optimization landscapes that are actually easier to navigate than the true simulator. Because the model learns to map actions directly to future states, it can smooth out messy intermediate dynamics—like a chaotic bounce or non-differentiable contact point—that would otherwise derail a gradient-based optimizer. The choice of architecture inherently changes the "temporal credit assignment," and Transformer AWMs naturally excel at assigning credit to actions taken far in the past.

The researchers evaluated their approach on several challenging benchmarks, including the chaotic Double Pendulum, a One-Bounce environment, and the Myriad Testbed (a suite of continuous real-world inspired MDPs with horizons up to 500 steps, such as optimizing chemotherapy dosages).

Performance comparison showing Final Return vs. Environment Length. Transformer AWMs maintain high performance over long horizons, while standard models and even simulator dynamics degrade significantly.

Key Results: * Chaotic Stability: In tasks like the double pendulum, standard world models and even the "true" simulator dynamics failed as the horizon grew. Transformer AWMs were the only models that maintained stable gradients and successfully solved the task. * Long-Horizon Superiority: On 500-step tasks in the Myriad benchmark, Transformer AWMs significantly outperformed Markovian models, HWMs, and model-free baselines like Soft Actor-Critic (SAC) and Online Decision Transformers. * Sample Efficiency: The proposed method achieved high performance with far fewer environment interactions compared to model-free alternatives.

Limitations and Future Work

While promising, the approach has limitations. The theoretical proofs and primary experiments focus heavily on deterministic environments. Although the authors introduce a "Random State Rollout" trick to help in stochastic settings, handling highly noisy real-world data requires further exploration. Additionally, the experiments utilized relatively low-dimensional state spaces; scaling AWMs to handle high-dimensional visual inputs (like raw pixels) remains an open challenge. Finally, in environments where the state changes significantly due to external factors not captured by the initial state, the "action-only" conditioning might eventually hit a representational bottleneck.

Despite these challenges, Action-conditioned World Models represent a significant step forward in stabilizing long-horizon policy gradients, proving that with the right architectural tweaks, Transformers can indeed conquer long-range RL.

Summary updated

Abstract

A natural approach for reinforcement learning is to predict future rewards by unrolling a neural network world model, and to backpropagate through the resulting computational graph to learn a control policy. However, this method often becomes impractical for long horizons, since typical world models induce hard-to-optimize loss landscapes. Transformers are known to efficiently propagate gradients over long horizons: could they be the solution to this problem? Surprisingly, we show that commonly-used transformer world models produce circuitous gradient paths, which can be detrimental to long-range policy gradients. To tackle this challenge, we propose a class of world models called Action-conditioned World Models (AWMs), designed to provide more direct routes for gradient propagation. We integrate such AWMs into a policy gradient framework that underscores the relationship between network architectures and the policy gradient updates they inherently represent. We demonstrate that AWMs can generate optimization landscapes that are easier to navigate even when compared to those from the simulator itself. This property allows transformer AWMs to produce better policies than competitive baselines in realistic long-horizon tasks.
BibTeX
@inproceedings{ma2024do,
  author = {Michel Ma and Tianwei Ni and Clement Gehring and Pierluca D'Oro and Pierre-Luc Bacon},
  title = {{Do Transformer World Models Give Better Policy Gradients?}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Uoved2xD81},
}
Figure