Towards Better Sample Efficiency in Multi-Agent Reinforcement Learning via Exploration

Amir Baghi, Jens Sjölund, Joakim Bergdahl, Linus Gisslén, Alessandro SestiniarXiv 2025 · arXiv · OpenReview

Abstract

Multi-agent reinforcement learning has shown promise in learning cooperative behaviors in team-based environments. However, such methods often demand extensive training time. For instance, the state-of-the-art method TiZero takes 40 days to train high-quality policies for a football environment. In this paper, we hypothesize that better exploration mechanisms can improve the sample efficiency of multi-agent methods. We propose two different approaches for better exploration in TiZero: a self-supervised intrinsic reward and a random network distillation bonus. Additionally, we introduce architectural modifications to the original algorithm to enhance TiZero's computational efficiency. We evaluate the sample efficiency of these approaches through extensive experiments. Our results show that random network distillation improves training sample efficiency by 18.8% compared to the original TiZero. Furthermore, we evaluate the qualitative behavior of the models produced by both variants against a heuristic AI, with the self-supervised reward encouraging possession and random network distillation leading to a more offensive performance. Our results highlights the applicability of our random network distillation variant in practical settings. Lastly, due to the nature of the proposed method, we acknowledge its use beyond football simulation, especially in environments with strong multi-agent and strategic aspects.

Introduction

Multi-Agent Reinforcement Learning (MARL) has demonstrated significant potential in mastering complex, team-based environments such as StarCraft II and football. However, these models are notoriously data-hungry. The current state-of-the-art football AI, TiZero, requires approximately 40 days of training across hundreds of processors to achieve high-quality performance. In industrial contexts like game development, this lengthy iteration loop represents a major bottleneck, preventing developers from quickly testing and refining agent behaviors.

The authors hypothesize that poor exploration is the primary cause of this inefficiency. Agents often spend excessive time repeating basic errors before discovering the complex strategies—such as coordinated passing and positioning—required for success.

Methodology

To address these challenges, the researchers introduced several enhancements to the TiZero framework, focusing on architectural efficiency and improved exploration.

Architectural Streamlining

The computationally intensive Long Short-Term Memory (LSTM) units were replaced with simpler, faster 4-layer Multi-layer Perceptrons (MLPs). Additionally, learned player ID encoders were replaced with fixed positional encodings, similar to those used in Transformer architectures, to provide a more stable way for agents to distinguish between teammates.

The modified TiZero architecture featuring MLP-based encoders and positional encodings.

Exploration Mechanisms

Two primary exploration techniques were evaluated:

  1. Random Network Distillation (RND): A "curiosity" bonus where a predictor network attempts to mimic a fixed, random target network. Prediction errors serve as an intrinsic reward, encouraging the agent to visit novel states.
  2. Self-supervised Intrinsic Reward (SSIR): This method learns a separate reward function based on the agent's own transitions, providing a denser signal to guide learning even when external rewards (like scoring a goal) are sparse.

Experimental Setup

The study utilized the Google Research Football (GRF) environment, a physics-based 3D simulation. Agents were trained using a curriculum of 10 scenarios with increasing difficulty, followed by a self-play phase. Performance was benchmarked against a Heuristic AI (HAI) provided by the GRF environment.

Results and Discussion

The integration of exploration bonuses significantly improved both the speed and stability of the training process.

Efficiency and Stability

The RND-augmented version of TiZero achieved an 18.8% improvement in sample efficiency over the baseline. Notably, TiZero-RND demonstrated superior stability; while the standard TiZero often experienced significant performance dips during training, the RND variant maintained a more consistent upward trajectory.

Win-rate comparison over environment steps, highlighting the stability of TiZero-RND.

Behavioral Shifts

The choice of exploration reward fundamentally altered the agents' playing styles. SSIR-trained agents developed a "possession-heavy" style, effectively keeping the ball but rarely attempting shots. In contrast, RND-trained agents were more offensive, taking more shots and performing more interceptions. This suggests that RND better encourages the discovery of high-value offensive actions.

Conclusion

By incorporating curiosity-driven exploration through Random Network Distillation, the researchers significantly reduced the training time required for complex MARL agents. The resulting agents were not only more efficient to train but also exhibited more sophisticated and stable behaviors. While this study focused on football simulation, the findings suggest that these techniques could be broadly applicable to other strategic multi-agent domains, such as autonomous driving or logistics.

Summary updated

Abstract

Multi-agent reinforcement learning has shown promise in learning cooperative behaviors in team-based environments. However, such methods often demand extensive training time. For instance, the state-of-the-art method TiZero takes 40 days to train high-quality policies for a football environment. In this paper, we hypothesize that better exploration mechanisms can improve the sample efficiency of multi-agent methods. We propose two different approaches for better exploration in TiZero: a self-supervised intrinsic reward and a random network distillation bonus. Additionally, we introduce architectural modifications to the original algorithm to enhance TiZero's computational efficiency. We evaluate the sample efficiency of these approaches through extensive experiments. Our results show that random network distillation improves training sample efficiency by 18.8% compared to the original TiZero. Furthermore, we evaluate the qualitative behavior of the models produced by both variants against a heuristic AI, with the self-supervised reward encouraging possession and random network distillation leading to a more offensive performance. Our results highlights the applicability of our random network distillation variant in practical settings. Lastly, due to the nature of the proposed method, we acknowledge its use beyond football simulation, especially in environments with strong multi-agent and strategic aspects.
BibTeX
@misc{baghi2025towards,
  author = {Amir Baghi and Jens Sjölund and Joakim Bergdahl and Linus Gisslén and Alessandro Sestini},
  title = {{Towards Better Sample Efficiency in Multi-Agent Reinforcement Learning via Exploration}},
  year = {2025},
  eprint = {2503.13077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.13077},
}
Figure