In traditional multi-agent reinforcement learning (MARL), agents are typically treated as independent actors who decide on their actions simultaneously. This creates a "circular dependency" problem: Agent A chooses an action based on what it thinks Agent B will do, while Agent B does the same for Agent A. As Gang Ding and colleagues describe it, this is like a flash mob where everyone wears noise-canceling headphones—without a way to see each other's immediate choices, agents often pick incompatible pieces of different successful plans, leading to coordination failure.
A growing body of research suggests a powerful alternative: treating multi-agent coordination as a sequence modeling problem. By decomposing a joint action into an autoregressive chain, we can transform a chaotic simultaneous game into a structured, sequential one.
The Core Intuition: Factoring the Joint Policy
The mathematical heart of this shift is the Multi-Agent Advantage Decomposition Theorem. Instead of trying to learn a massive joint distribution of all possible action combinations—which grows exponentially with the number of agents—researchers like Wei Fu and Zongqing Lu propose factorizing the joint policy $\pi(\mathbf{a}|s)$ into a sequence of conditional distributions:
$$\pi(a_1, a_2, \dots, a_n | s) = \pi(a_1|s) \cdot \pi(a_2|s, a_1) \dots \pi(a_n|s, a_1, \dots, a_{n-1})$$
In this framework, agents act in a specific order within a single time step. Agent 1 acts first; Agent 2 observes Agent 1’s choice and then acts; Agent 3 observes both, and so on. This sequential dependency allows downstream agents to specialize their behavior based on upstream choices, effectively capturing hierarchical dependencies that independent policies cannot represent.
The Transformer as a Coordination Engine
The architecture of choice for this sequential decomposition is the Transformer. In the Multi-Agent Transformer (MAT), an encoder processes the global environment state, while an autoregressive decoder generates agent actions one by one.
By using masked self-attention, the decoder ensures that each agent's policy is conditioned on the decisions of all previously processed agents. This mechanism, also explored in the Stackelberg Decision Transformer (STEER) by Bin Zhang, frames multi-agent interaction as a leader-follower hierarchy. The "leader" (the first agent in the sequence) sets the tone, and the "followers" optimize their responses accordingly.
Why It Works: Multi-Modality and Consistency
This approach solves three critical problems in multi-agent systems:
- Multi-modal Coordination: In environments with multiple ways to succeed (e.g., two agents can both go left or both go right), independent agents might "split the difference" and collide. Autoregressive models, as noted by Dan Qiao in the OMSD framework, capture these multi-modal dependencies, ensuring the team commits to a single coherent strategy.
- Offline Policy Mismatch: In offline RL, agents learn from datasets containing many different strategies. Yihe Zhou and Claude Formanek (in the Oryx algorithm) show that sequential conditioning prevents "policy mismatch," where agents accidentally mix-and-match actions from different trajectories in the training data.
- Conflict Resolution: In logistics tasks like warehouse routing, Zeyuan Zhao demonstrates that autoregressive policies allow agents to implicitly exchange information. If Agent A claims a specific path, Agent B sees that commitment in the sequence and automatically chooses a non-conflicting route.
Motion as Language
The most striking application of this idea is in traffic simulation and motion forecasting. Models like MotionLM by Ari Seff and Trajeglish by Jonah Philion treat the movement of vehicles and pedestrians as a "language" of motion.
By discretizing continuous trajectories into "motion tokens" and using a GPT-style transformer for next-token prediction, these models generate multi-agent futures that are temporally and spatially consistent. Instead of predicting all car movements at once, they predict them sequentially within each time step. This intra-timestep conditioning allows the model to resolve dense traffic conflicts—like two cars merging into the same lane—by letting one car "speak" its intention first.
Scaling the Sequence
While sequential decoding sounds slow, new architectures are making it scalable. Omayma Mahjoub and the team behind Sable and Oryx have replaced standard attention with more efficient mechanisms like "retention," allowing these autoregressive models to scale to systems with over 1,000 agents.
By moving from simultaneous "guessing" to sequential "responding," this research direction is turning the problem of multi-agent coordination into a solved problem of sequence modeling.