Abstract
Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions. In this work, we propose State-Action-Reward Transformer (StARformer) for visual RL, which explicitly models short-term state-action-reward representations (StAR-representations), essentially introducing a Markovian-like inductive bias to improve long-term modeling. Our approach first extracts StAR-representations by self-attending image state patches, action, and reward tokens within a short temporal window. These are then combined with pure image state representations -- extracted as convolutional features, to perform self-attention over the whole sequence. Our experiments show that StARformer outperforms the state-of-the-art Transformer-based method on image-based Atari and DeepMind Control Suite benchmarks, in both offline-RL and imitation learning settings. StARformer is also more compliant with longer sequences of inputs. Our code is available at https://github.com/elicassion/StARformer.
Introduction
Recent breakthroughs have reframed Reinforcement Learning (RL) as a sequence modeling problem, utilizing Transformers to predict actions based on historical trajectories of states, actions, and rewards. However, standard Transformer architectures typically treat all tokens in a sequence uniformly. This approach requires the model to learn fundamental environmental dynamics—such as the Markov property—entirely from scratch, which can be inefficient.
In RL environments, the immediate past often exerts a significantly stronger influence on the next action than the distant past. Naive Transformers can suffer from "information dilution," where critical local dependencies are obscured by the noise of a long history. Furthermore, standard models frequently compress image states into global features, potentially losing fine-grained spatial details—such as the precise relative positions of objects—that are vital for effective control.
Methodology
StARformer addresses these challenges by explicitly modeling short-term "State-Action-Reward" (StAR) transitions through an interleaved architecture consisting of two primary components:
- Step Transformer: This module focuses on a "local" temporal window. It processes a single-step transition—comprising the previous action ($a_{t-1}$), previous reward ($r_{t-1}$), and current state ($s_t$). By utilizing ViT-style image patches for the state, the Step Transformer learns fine-grained spatial relations, such as object movement and interaction.
- Sequence Transformer: This module manages the "global" trajectory. It integrates the compressed "StAR-representations" from the Step Transformer with global image features extracted via a standard CNN. It then performs self-attention across the entire sequence to capture long-term strategic patterns.
These components are connected layer-wise, allowing local and global information to be fused at multiple levels of abstraction. This design introduces a Markovian Inductive Bias, relieving the Sequence Transformer from re-learning basic physics and allowing it to focus on high-level strategy. A key finding is the "Patch + Conv" hybrid: local patches are superior for modeling single-step transitions, while global convolutional features are better for long-range sequence modeling.
Experimental Results
StARformer was evaluated on the Atari benchmark (six games) and the DeepMind Control Suite (DMC) (three tasks) in both offline RL and imitation learning settings.
Performance in Offline RL
In offline RL, StARformer demonstrated substantial improvements over the baseline Decision Transformer (DT). For instance, in the Atari game Breakout, StARformer achieved a score of 124.1, significantly higher than DT's 76.9. In the DMC Cheetah task, the performance gap was even more pronounced, with StARformer scoring 275.3 against DT's 27.9.
Scaling and Robustness
A key advantage of StARformer is its ability to scale with history length. While DT's performance often plateaus or degrades with longer sequences (e.g., $T=30$), StARformer’s performance continues to improve as the sequence length increases. Additionally, the model shows robustness to reward framing, performing consistently well even with simple stepwise rewards where DT typically relies heavily on "Return-to-Go" targets.
Visual Interpretability
Analysis of the model's attention maps reveals that the Step Transformer effectively learns to track task-relevant objects. In Breakout, the attention shifts dynamically to the paddle and ball, providing a clear visual rationale for the predicted actions.
Conclusion and Future Work
StARformer introduces a novel way to incorporate local dynamics into sequence-based RL. By interleaving local step-wise modeling with global sequence modeling, it achieves state-of-the-art performance on visual RL benchmarks. While the architecture introduces additional complexity, its superior scaling properties and interpretability make it a powerful tool for complex control tasks. Future research may explore applying this architecture to even longer sequences or more complex multi-modal environments where local and global dependencies are even more distinct.
Summary updated