Selective Learning for Sample-Efficient Training in Multi-Agent Sparse Reward Tasks (Extended Abstract)

Xinning Chen, Xuan Liu, Yanwen Ba, Shigeng Zhang, Bo Ding, Kenli LiIJCAI 2024

Abstract

Multi-Agent Reinforcement Learning (MARL) in sparse reward environments faces significant challenges due to the curse of dimensionality and the difficulty of credit assignment. We present Multi-Agent Selective Learning (MASL), a framework that utilizes a centralized backtracking model to generate "recall traces" from successful outcomes and an attention-based selector to focus on relevant agent interactions. Our results on cooperative benchmarks demonstrate that MASL significantly improves sample efficiency and exhibits robust zero-shot scalability, outperforming several state-of-the-art baselines.

Introduction

Reinforcement Learning (RL) typically relies on frequent feedback to optimize agent policies. However, in real-world multi-agent scenarios such as autonomous driving or swarm robotics, rewards are often sparse. Agents may execute thousands of actions before receiving a single success signal. In MARL, this sparsity is exacerbated by the exponential growth of the joint state-action space and the non-stationarity of the environment as multiple agents learn simultaneously. Furthermore, assigning credit for a team success to individual agent actions remains a complex hurdle. Existing experience-sharing methods often suffer from information overload, where high-value states are obscured by irrelevant data.

Methodology

To address these challenges, we propose the Multi-Agent Selective Learning (MASL) framework, which incorporates three primary mechanisms into the Centralized Training with Decentralized Execution (CTDE) paradigm:

  1. Centralized Backtracking Model ($B_\phi$): Instead of relying solely on forward exploration, MASL employs a backtracking model to generate "recall traces"—imagined sequences of state-action pairs that lead to identified high-value states. These traces serve as a guided map, allowing agents to bypass aimless exploration and focus on known successful trajectories.
  2. Key Information Selector: To mitigate the curse of dimensionality, we implement a selector using a scaled dot-product attention mechanism. This component calculates relevancy weights between agents, allowing each agent to focus only on the top $K$ most relevant neighbors. This selective attention filters out noise and stabilizes the training process.
  3. Retrogression-based Selection: This method identifies specific agent trajectories that contributed to team rewards, ensuring that the backtracking model is trained on the most effective experiences.
Figure 1: Architecture of the MASL framework, illustrating the interaction between the Backtracking Model and the selective attention mechanism.

Experimental Evaluation

We evaluated MASL using modified Multi-Particle Environment (MPE) benchmarks, focusing on two cooperative tasks:

  • Resource Collection (RC): $N$ agents must gather $L$ resources while avoiding collisions.
  • Rover Exploration (RE): Agents must cooperatively explore target areas, with a coupling requirement that necessitates multiple agents being present at a target simultaneously to receive a reward.

Experiments were conducted with varying scales, ranging from 5 to 20 agents.

Results and Discussion

MASL demonstrated superior sample efficiency across all tested scenarios, reaching higher reward plateaus faster than baselines such as MADDPG, MAAC, and Mean Field (MF) reinforcement learning. In the 20-agent Resource Collection task, MASL achieved a 30.2% improvement in sample efficiency over MADDPG-MD.

Figure 4: (a) Visualization of the Rover Exploration task. (b-d) Reward curves for RE-6, RE-8, and RE-12, demonstrating MASL's performance advantages.

A key finding is MASL's zero-shot scalability. Policies trained on 5 agents were successfully deployed in environments with 30 agents, achieving a 36% completion rate, significantly higher than the 24.7% achieved by the next best baseline. The computational overhead for the backtracking and selection mechanisms remained low, at approximately 0.93% of total computation time.

Conclusion

MASL effectively addresses the "needle in a haystack" problem of sparse rewards in MARL by prioritizing high-quality experience sharing and selective attention. Future research will explore the application of MASL in high-fidelity 3D simulations and its performance with heterogeneous agent teams.

Summary updated

BibTeX
@inproceedings{chen2024selective,
  author = {Xinning Chen and Xuan Liu and Yanwen Ba and Shigeng Zhang and Bo Ding and Kenli Li},
  title = {{Selective Learning for Sample-Efficient Training in Multi-Agent Sparse Reward Tasks (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure