Sequential Order Adjustment of Action Decisions for Multi-Agent Transformer (Student Abstract)

Shota Takayama, Katsuhide FujitaAAAI 2025 · DOI

TL;DR

This paper introduces AOAD-MAT, a novel multi-agent reinforcement learning (MARL) model that dynamically learns the optimal order in which agents should make decisions. By treating action decision-making as a sequence prediction task, AOAD-MAT significantly outperforms existing methods on complex cooperative tasks, demonstrating the crucial role of action order in MARL performance.

What Problem Does This Paper Address?

Multi-agent reinforcement learning (MARL) aims to train multiple agents to cooperate in shared environments. While recent advancements, particularly the Multi-Agent Transformer (MAT), have improved performance by modeling agent interactions as sequences, they overlook a critical factor: the order in which agents make their decisions. In MARL, the sequence of actions can profoundly impact overall team performance and stability. This paper addresses this gap by proposing a method that explicitly learns and optimizes this action decision order.

Methods

The core innovation is the AOAD-MAT (Agent Order of Action Decisions-MAT) model. It builds upon the existing MAT architecture but introduces a novel mechanism within the decoder (actor network). This mechanism performs a "branching subtask" to predict the next agent that should act in the sequence.

Key components:

  • Sequential Action Decision Order Prediction: A learner integrated into the decoder predicts the next agent to act, complementing the standard action prediction.
  • Transformer-based Actor-Critic: Leverages the Transformer architecture for its ability to model sequential dependencies.
  • Modified Loss Function: The decoder acts as a multitask learner, predicting both the current agent's action and the next agent in the sequence. The loss function is designed to converge both these predictions. Instead of a weighted sum of losses (common in multitask learning), it uses a product of ratios of probability distributions for action and next-agent predictions within the PPO framework.
  • Dynamic Swapping: Based on the predicted next agent, the latent representations of observations are swapped, ensuring the correct agent's information is processed at each step.

Datasets

The paper evaluates AOAD-MAT on two benchmark environments:

  • SMAC (StarCraft Multi-Agent Challenge): A suite of challenging cooperative tasks in the StarCraft game, featuring discrete action spaces. Specific scenarios tested include 5m_vs_6m, MMM2, 6h_vs_8z, and 3s5z vs 3a6z.
  • MA-MuJoCo: A multi-agent extension of the MuJoCo physics simulator, used here with continuous action spaces. The specific task evaluated is controlling a six-jointed HalfCheetah robot to maximize forward velocity.

Key Insights and Ideas

  • Action Order Matters Significantly: The most striking insight is that the sequence in which agents act is not arbitrary but a critical determinant of overall performance in MARL.
  • Dynamic Order Adaptation: Learning the optimal order allows agents to adapt their decision-making process dynamically, which is particularly beneficial in complex, high-dimensional state spaces.
  • Improved Exploration: The adaptive action order can help navigate vast exploration spaces more effectively, leading to better learning outcomes.
  • Synergy with Transformer Architecture: The Transformer's inherent ability to handle sequences makes it a natural fit for modeling and optimizing the action decision order.

Key Results

The proposed AOAD-MAT model demonstrated superior performance compared to baseline methods, including the original MAT and MAT-adjust:

  • SMAC: Achieved 100% win rates in several scenarios (5m_vs_6m, MMM2, 6h_vs_8z, 3s5z vs 3a6z), matching or exceeding the performance of baselines, often with fewer steps or better stability (indicated by lower standard deviations).
  • MA-MuJoCo (HalfCheetah): Showed approximately a 10% improvement in median reward compared to baselines, with a notable increase in the upper bound of the 95% confidence interval, suggesting higher peak performance.

Limitations and Future Work

  • Limitations: The paper focuses on a student abstract, so detailed limitations are not extensively discussed. However, the reliance on the Transformer architecture might imply computational costs for very large numbers of agents. The initial agent to act is still predetermined, which could be a point for future exploration.
  • Future Work: The authors suggest that their work highlights the effectiveness of adjusting agent order in MARL. Future research could explore more sophisticated methods for determining the initial agent, or investigate how this learned order interacts with other aspects of agent coordination and communication.

Summary updated

BibTeX
@inproceedings{takayama2025sequential,
  author = {Shota Takayama and Katsuhide Fujita},
  title = {{Sequential Order Adjustment of Action Decisions for Multi-Agent Transformer (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35306},
}
Figure