Abstract
The difficulty of appropriately assigning credit is particularly heightened in cooperative multi-agent reinforcement learning (MARL) with sparse rewards. While Automatic Subgoal Generation (ASG) has emerged as a viable approach, end-to-end learning of complex task planning often requires massive training samples and can generate spurious subgoals. We propose Semantically Aligned task decomposition in MARL (SAMA), a method that prompts pretrained language models (PLMs) with chain-of-thought to suggest potential goals, provide suitable goal decomposition, and perform self-reflection-based replanning. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods on challenging sparse-reward tasks like Overcooked and MiniRTS.
Introduction
In cooperative MARL, agents must overcome two primary credit assignment hurdles: temporal (identifying which action in a sequence led to success) and structural (determining which agent was responsible). Sparse rewards exacerbate these issues, as agents often fail to find successful trajectories through random exploration. Traditional ASG methods attempt to bridge this gap but often suffer from sample inefficiency due to the generation of irrelevant subgoals. SAMA addresses this by injecting "common-sense" priors from PLMs into the decision-making process.
The SAMA Framework
SAMA treats task planning as a "disentangled" process guided by natural language. By using PLMs like GPT-4, the framework prunes the exploration space, focusing agents on logically sound objectives.
The framework operates through four key stages:
- Automated Preprocessing: PLMs generate task manuals from environment source code or documentation, making the system environment-agnostic.
- Semantically Aligned Decomposition: Using Chain-of-Thought prompting, the system decomposes high-level goals into specific, disjointed subgoals for each agent.
- Language-Grounded MARL: Agents are trained with goal-conditioned policies to interpret and execute natural language instructions (e.g., "Chef A: take an onion to the bar").
- Self-Reflection Mechanism: A feedback loop allows the PLM to analyze failures and adjust plans based on environmental feedback, correcting for potential hallucinations or logical errors.
Experimental Results
SAMA was evaluated on several challenging benchmarks, including Overcooked-AI and MiniRTS. Overcooked-AI layouts, such as Forced Coordination and Counter Circuit, require tight coordination and long-term planning.
SAMA demonstrated superior sample efficiency, reaching near-SOTA performance in the Forced Coordination layout using only 10% of the training samples required by traditional end-to-end algorithms. As shown in the comparative results, SAMA consistently outperformed existing subgoal-based methods (like MASER and LDSA) and remained competitive with specialized SOTA models across diverse layouts.
Discussion and Conclusion
SAMA highlights the utility of PLMs as a pruning tool for reinforcement learning. By forcing disentangled decision-making, the framework naturally resolves structural credit assignment. However, limitations exist, such as reasoning conflicts when PLM priors differ from specific game mechanics and the computational cost of high-end model inference. Future work may integrate vision-language models to streamline state translation and further enhance generalizability.
Summary updated