Multi-Agent Credit Assignment with Pretrained Language Models

Wenhao Li, Dan Qiao, Baoxiang Wang, Xiangfeng Wang, Wei Yin, Hao Shen, Bo Jin, Hongyuan ZhaAISTATS 2025 · OpenReview

Abstract

The difficulty of appropriately assigning credit is particularly heightened in cooperative multi-agent reinforcement learning (MARL) with sparse rewards. While Automatic Subgoal Generation (ASG) has emerged as a viable approach, end-to-end learning of complex task planning often requires massive training samples and can generate spurious subgoals. We propose Semantically Aligned task decomposition in MARL (SAMA), a method that prompts pretrained language models (PLMs) with chain-of-thought to suggest potential goals, provide suitable goal decomposition, and perform self-reflection-based replanning. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods on challenging sparse-reward tasks like Overcooked and MiniRTS.

Introduction

In cooperative MARL, agents must overcome two primary credit assignment hurdles: temporal (identifying which action in a sequence led to success) and structural (determining which agent was responsible). Sparse rewards exacerbate these issues, as agents often fail to find successful trajectories through random exploration. Traditional ASG methods attempt to bridge this gap but often suffer from sample inefficiency due to the generation of irrelevant subgoals. SAMA addresses this by injecting "common-sense" priors from PLMs into the decision-making process.

The SAMA Framework

SAMA treats task planning as a "disentangled" process guided by natural language. By using PLMs like GPT-4, the framework prunes the exploration space, focusing agents on logically sound objectives.

The SAMA framework overview, illustrating the pipeline from goal generation to self-reflection-based replanning.

The framework operates through four key stages:

  1. Automated Preprocessing: PLMs generate task manuals from environment source code or documentation, making the system environment-agnostic.
  2. Semantically Aligned Decomposition: Using Chain-of-Thought prompting, the system decomposes high-level goals into specific, disjointed subgoals for each agent.
  3. Language-Grounded MARL: Agents are trained with goal-conditioned policies to interpret and execute natural language instructions (e.g., "Chef A: take an onion to the bar").
  4. Self-Reflection Mechanism: A feedback loop allows the PLM to analyze failures and adjust plans based on environmental feedback, correcting for potential hallucinations or logical errors.

Experimental Results

SAMA was evaluated on several challenging benchmarks, including Overcooked-AI and MiniRTS. Overcooked-AI layouts, such as Forced Coordination and Counter Circuit, require tight coordination and long-term planning.

Comparative performance of SAMA against state-of-the-art and subgoal-based MARL methods across various Overcooked-AI layouts.

SAMA demonstrated superior sample efficiency, reaching near-SOTA performance in the Forced Coordination layout using only 10% of the training samples required by traditional end-to-end algorithms. As shown in the comparative results, SAMA consistently outperformed existing subgoal-based methods (like MASER and LDSA) and remained competitive with specialized SOTA models across diverse layouts.

Discussion and Conclusion

SAMA highlights the utility of PLMs as a pruning tool for reinforcement learning. By forcing disentangled decision-making, the framework naturally resolves structural credit assignment. However, limitations exist, such as reasoning conflicts when PLM priors differ from specific game mechanics and the computational cost of high-end model inference. Future work may integrate vision-language models to streamline state translation and further enhance generalizability.

Summary updated

Abstract

The difficulty of appropriately assigning credit is particularly heightened in cooperative MARL with sparse reward, due to the concurrent time and structural scales involved. Automatic subgoal generation (ASG) has recently emerged as a viable MARL approach inspired by utilizing subgoals in intrinsically motivated reinforcement learning. However, end-to-end learning of complex task planning from sparse rewards without prior knowledge, undoubtedly requires massive training samples. Moreover, the diversity-promoting nature of existing ASG methods can lead to the "over-representation" of subgoals, generating numerous spurious subgoals of limited relevance to the actual task reward and thus decreasing the sample efficiency of the algorithm. To address this problem and inspired by the disentangled representation learning, we propose a novel "disentangled" decision-making method, Semantically Aligned task decomposition in MARL (SAMA), that prompts pretrained language models with chain-of-thought that can suggest potential goals, provide suitable goal decomposition and subgoal allocation as well as self-reflection-based replanning. Additionally, SAMA incorporates language-grounded MARL to train each agent’s subgoal-conditioned policy. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods, as evidenced by its performance on two challenging sparse-reward tasks, Overcooked and MiniRTS. The code is available at \url{https://anonymous.4open.science/r/SAMA/.}
BibTeX
@inproceedings{li2025multi,
  author = {Wenhao Li and Dan Qiao and Baoxiang Wang and Xiangfeng Wang and Wei Yin and Hao Shen and Bo Jin and Hongyuan Zha},
  title = {{Multi-Agent Credit Assignment with Pretrained Language Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=ZRPzlxrrIz},
}
Figure