Model-based reinforcement learning (RL) often relies on training a "world model" to mimic the environment. To improve a control policy, this model is unrolled into the future, and the reward signal is backpropagated through the model's predictions to update the policy's parameters. However, this approach frequently breaks down over long horizons. In complex or chaotic systems, the optimization landscape becomes highly irregular, causing gradients to either vanish to zero or explode to infinity.
Given that Transformers have revolutionized long-range dependency handling in sequence modeling, they seem like a natural solution. Yet, simply plugging Transformers in as world models hasn't yielded the expected breakthrough in RL. A recent paper presented at ICML 2024 investigates why this happens and proposes a structural fix to unlock the true potential of Transformers in long-horizon RL.
The Flaw in Standard World Models
The authors identify a fundamental structural flaw in how world models are typically deployed. Most standard models are History-conditioned World Models (HWMs) or Markovian models. To predict the next state, they autoregressively take the previously predicted state as an input.
While this makes intuitive sense for forward prediction, it creates a massive problem for backpropagation. When calculating policy gradients, this autoregressive loop forces the gradients to travel through "circuitous paths," compounding errors and gradient instabilities exponentially over time. Even a perfect Transformer becomes effectively recurrent—and thus unstable—if its output state is continuously fed back in as an input.
The Solution: Action-Conditioned World Models (AWMs)
To resolve this, the researchers propose switching to Action-conditioned World Models (AWMs).
Instead of predicting the next state based on the immediate prior state ($\hat{s}_{t+1} = f(\hat{s}_t, a_t)$), an AWM predicts future states using only the initial state and the sequence of actions taken so far: $\hat{s}_{t+1} = g(s_1, a_1, \dots, a_t)$.
By conditioning only on the initial state and the action sequence, the gradient flows directly from the reward through the Transformer's attention mechanism straight to the actions. The authors prove theoretically that while standard models suffer from exponential gradient growth ($O(L_s^H)$), a Transformer-based AWM exhibits a much more manageable polynomial growth ($O(H^3)$).
Key Insights and Performance
Surprisingly, AWMs can produce optimization landscapes that are actually easier to navigate than the true simulator. Because the model learns to map actions directly to future states, it can smooth out messy intermediate dynamics—like a chaotic bounce or non-differentiable contact point—that would otherwise derail a gradient-based optimizer. The choice of architecture inherently changes the "temporal credit assignment," and Transformer AWMs naturally excel at assigning credit to actions taken far in the past.
The researchers evaluated their approach on several challenging benchmarks, including the chaotic Double Pendulum, a One-Bounce environment, and the Myriad Testbed (a suite of continuous real-world inspired MDPs with horizons up to 500 steps, such as optimizing chemotherapy dosages).
Key Results: * Chaotic Stability: In tasks like the double pendulum, standard world models and even the "true" simulator dynamics failed as the horizon grew. Transformer AWMs were the only models that maintained stable gradients and successfully solved the task. * Long-Horizon Superiority: On 500-step tasks in the Myriad benchmark, Transformer AWMs significantly outperformed Markovian models, HWMs, and model-free baselines like Soft Actor-Critic (SAC) and Online Decision Transformers. * Sample Efficiency: The proposed method achieved high performance with far fewer environment interactions compared to model-free alternatives.
Limitations and Future Work
While promising, the approach has limitations. The theoretical proofs and primary experiments focus heavily on deterministic environments. Although the authors introduce a "Random State Rollout" trick to help in stochastic settings, handling highly noisy real-world data requires further exploration. Additionally, the experiments utilized relatively low-dimensional state spaces; scaling AWMs to handle high-dimensional visual inputs (like raw pixels) remains an open challenge. Finally, in environments where the state changes significantly due to external factors not captured by the initial state, the "action-only" conditioning might eventually hit a representational bottleneck.
Despite these challenges, Action-conditioned World Models represent a significant step forward in stabilizing long-horizon policy gradients, proving that with the right architectural tweaks, Transformers can indeed conquer long-range RL.
Summary updated