Traditional reinforcement learning (RL) views decision-making through the lens of the Markov Decision Process (MDP). In this classical framework, agents learn value functions or policy gradients by interacting with an environment, updating their parameters via temporal difference (TD) learning or policy optimization. While elegant, bootstrapping over long horizons often introduces training instability, optimization bottlenecks, and severe credit assignment challenges.
A powerful alternative paradigm has emerged: treating sequential decision-making entirely as an autoregressive sequence modeling problem. Instead of explicitly computing value functions or optimizing policy gradients, this approach flattens a history of states, actions, and rewards into a unified sequence of tokens. By training a causally masked transformer on these sequences, policy learning reduces to next-token prediction—the exact same objective that drives modern Large Language Models (LLMs).
As argued by Xiaoqian Liu et al. in Position: Foundation Agents as the Paradigm Shift for Decision Making, this shift unifies perception, reasoning, and action into a single scalable architecture, paving the way for general-purpose foundation agents that scale predictably with data and compute.
The Mechanics of Trajectory Modeling
To understand why this paradigm works, consider how a trajectory is presented to a transformer. Instead of processing isolated state transitions $(s_t, a_t, s_{t+1})$, a trajectory $\tau$ is interleaved as a continuous sequence of target goals (or returns-to-go), states, and actions:
$$\tau = (\dots, R_t, s_t, a_t, R_{t+1}, s_{t+1}, a_{t+1}, \dots)$$
During training, a causal attention mask ensures that the prediction of the next token can only depend on past tokens. When generating an action $a_t$, the model attends to the historical context of states and actions, conditioned on the desired target return $R_t$.
This formulation completely bypasses the need for dynamic programming or value iteration. For instance, Ekin Akyürek et al. demonstrate in Pre-Trained Language Models for Interactive Decision-Making that framing interactive decision-making as a sequence-to-sequence task allows transformers to leverage combinatorial generalization, mapping complex historical contexts directly to discrete action spaces.
Even highly strategic domains can be cast into this format. Xidong Feng et al. show in ChessGPT: Bridging Policy Learning and Language Modeling that representing board states and game moves as standardized text strings enables a transformer to learn complex strategic behaviors purely through next-token prediction objectives, effectively bridging the gap between raw policy execution and textual reasoning.
Why Causal Attention Succeeds
Why does next-token prediction succeed where traditional RL often struggles? The intuition lies in how self-attention mechanisms handle context, credit assignment, and generalization.
1. Mitigating Forgetting over Long Horizons
Traditional recurrent networks (like LSTMs) compress history into a fixed-size hidden state, which suffers from vanishing gradients over long horizons. Markovian policies, on the other hand, assume the current state contains all necessary information, failing when environments are partially observable.
Causal transformers solve this by providing a direct, uncompressed computational path between any two time steps in the context window. As noted by Shuo Wang et al. in What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction, transformers are uniquely suited for multi-step trajectory modeling because they do not suffer from the exponential decay of early state information. This allows the model to maintain a consistent context during multi-step return estimation, preventing the loss of critical initial observations.
2. In-Context System Identification
By feeding a history of states and actions into a causal transformer, the model can implicitly infer latent properties of the environment—such as system dynamics, friction, or agent intentions—directly from the context window.
Weinan Zhang et al. exploit this property in Multi-embodiment Legged Robot Control as a Sequence Modeling Problem to train a single controller capable of managing legged robots with vastly different physical proportions. Instead of relying on explicit system identification, the transformer uses the temporal sequence to infer the robot's specific morphology in real time, adapting its motor commands dynamically.
3. Scaling to Physical Control
This sequence prediction capability scales directly to complex real-world physical systems. Ilija Radosavovic demonstrates in Humanoid Locomotion as Next Token Prediction that full-scale humanoid locomotion can be mastered by discretizing sensorimotor trajectories into tokens and modeling them autoregressively. By treating physical control as a next-token prediction task, the policy successfully transfers zero-shot from simulation and video data to real-world city streets.
Generative World Models
The sequence modeling paradigm extends beyond policy generation; it can also be used to learn the transition dynamics of the environment itself. By treating future states, rewards, and termination signals as tokens to be predicted, transformers act as highly sample-efficient world models.
[ Historical Context ] ---> [ Causal Transformer ] ---> [ Predicted Future ]
(States, Actions, Goals) (Self-Attention) (Next State, Reward)
In Transformers are Sample-Efficient World Models, Vincent Micheli et al. present an agent trained entirely within an "imagined" world simulated by an autoregressive transformer. Because the transformer captures long-horizon consistency far better than traditional recurrent architectures, the agent can successfully optimize its policy inside the simulation, achieving exceptional sample efficiency on complex benchmarks without real-world lookahead search.
Open Challenges
Despite its empirical success, treating decision-making as sequence prediction introduces new theoretical and practical hurdles. As Xiaoqian Liu et al. point out, the field currently lacks a comprehensive mathematical framework equivalent to the Bellman equations to explain why autoregressive trajectory modeling generalizes so effectively across non-Markovian settings.
Furthermore, because the model generates trajectories token-by-token, it is susceptible to error compounding over long horizons. If a model predicts a slightly suboptimal action or state, that prediction enters its own context window, potentially leading to a cascade of compounding errors. Mitigating this drift without sacrificing the simplicity of the next-token prediction objective remains an active area of research.