Abstract: While task generalisation is widely studied in the context of single-agent reinforcement learning (RL), little research exists in the context of multi-agent RL. The research that does exist usually considers task generalisation implicitly as a part of the environment, and when it is considered explicitly there are no theoretical guarantees. We propose Goal-Oriented Learning for Multi-Task Multi-Agent RL (GOLeMM), a method that achieves provably optimal task generalisation that, to the best of our knowledge, has not been achieved before in multi-agent RL (MARL). After learning an optimal goal-oriented value function for a single arbitrary task, our method can zero-shot infer the optimal policy for any other task in the distribution—given only knowledge of the terminal rewards for each agent for the new task and learnt task. Empirically in a tabular domain we show that our method is able to generalise over a full task distribution, while representative baselines are only able to learn a small subset of the task distribution—given the same knowledge about tasks. Additionally, while leveraging function approximation we demonstrate our method in a high-dimensional continuous domain and obtain superior task generalisation than a representative baseline.
Introduction
In Multi-Agent Reinforcement Learning (MARL), agents typically learn to solve specific tasks through extensive trial and error. However, when task parameters change—such as robots in a warehouse being reassigned to different shelves—agents often must restart the learning process from scratch. While single-agent RL has made significant strides in generalization, MARL has lagged behind. Current MARL generalization often occurs implicitly through deep learning, lacking theoretical guarantees and requiring vast amounts of data. We address this gap by introducing a framework that proves a multi-agent system can optimally solve new tasks without further training.
Methodology: GOLeMM
GOLeMM (Goal-Oriented Learning for Multi-Task Multi-Agent RL) is built upon three primary innovations:
- Multi-Agent World Value Functions (WVFs): Unlike standard value functions that focus on immediate rewards, GOLeMM learns a WVF. This function encodes the joint capability of agents to reach any possible combination of goals within the environment.
- Zero-Shot Inference: The WVF captures the underlying dynamics and coordination requirements of the environment. We prove that the optimal policy for a new task can be mathematically derived by plugging the new task's terminal rewards into the previously learned WVF.
- Symmetry and Efficiency: To mitigate the "curse of dimensionality," GOLeMM leverages agent-level termination (agents stop affecting rewards once a goal is reached) and homogeneity (identical agents share value representations).
Experimental Evaluation
We evaluated GOLeMM in two distinct environments:
- Tabular Grid World: A 2D warehouse-inspired grid where two agents navigate to specific goal squares while avoiding collisions. This environment serves to validate the theoretical optimality of our method.
- Multi-Task Multi-Particle Environment (MT-MPE): A high-dimensional, continuous-space environment requiring function approximation via neural networks. It involves complex physics, including velocity and collision dynamics.
Key Results and Insights
The core insight of GOLeMM is the separation of environment "physics" from specific "goals." By learning the "World" (WVF) rather than a specific "Task," agents gain a fundamental understanding of the environment that makes specific reward structures secondary.
- Total Generalization: In the tabular domain, GOLeMM successfully inferred the optimal policy for 99.81% of all possible tasks in the distribution after learning just one.
- Efficiency: While GOLeMM requires a higher upfront cost to learn the initial WVF, it scales significantly better than baselines like Centralized Q-Learning (CQL) or Independent Q-Learning (IQL) as the number of tasks increases.
- Continuous Space Performance: In the complex MT-MPE environment, GOLeMM significantly outperformed Universal Value Function Approximators (UVFA), achieving higher average returns and faster convergence.
Conclusion and Future Work
GOLeMM represents the first method to achieve provably optimal task generalization in cooperative MARL. However, several avenues for future research remain. Currently, the method relies on a centralized controller, which may face scaling challenges with very large numbers of agents. Additionally, the current proofs assume goal-reaching episodic tasks and full observability. Future work will explore extending GOLeMM to broader task distributions and partially observable settings.
Summary updated