rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer

Felipe B. Martins, Mateus G. Machado, Hansenclever F. Bassani, Pedro H. M. Braga, Edna BarrosRobocup 2021 · DOI

TL;DR

This paper introduces rSoccer, an open-source framework and simulator for training reinforcement learning (RL) agents in robot soccer. It addresses the challenge of applying RL to robotics by providing standardized environments and benchmarks for both single and multi-agent robot soccer tasks, aiming to foster reproducible research in this domain.

What Problem Does This Paper Address?

Reinforcement learning has shown impressive results in complex decision-making tasks like game playing. However, applying RL to real-world robotics, especially in dynamic environments like robot soccer, is challenging. Key issues include the high sample complexity required for training, the difficulty of transferring learned policies from simulation to reality (sim-to-real), and a lack of standardized, reproducible environments for research. Existing robot soccer simulators often lack extensibility, proper real-world simulation fidelity, or are not easily adaptable for RL research, hindering progress and reproducibility in the field. This paper aims to bridge this gap by providing a comprehensive framework for studying RL in robot soccer.

Methods

The core of the rSoccer framework is built around three modules:

  • Simulator: A custom simulator named rSim is developed, based on the grSim simulator but optimized for RL. Key modifications include removing graphical interfaces for performance, enabling synchronous operation for consistent training, supporting a variable number of robots, and providing direct Python calls for faster communication.
  • Environment: This module implements the OpenAI Gym interface, allowing seamless integration with standard RL algorithms. It handles communication between the agent and the simulator, defines observations and rewards, and manages the environment's state. The framework provides an API for creating new environments, requiring the implementation of only four key methods.
  • Render: A separate module for visualizing the simulation without impacting performance, useful for monitoring training progress.

The framework defines standardized data structures called "entities" (Ball, Robot, Frame, Field) to ensure consistent communication across modules.

Datasets

The paper does not explicitly list traditional datasets. Instead, it introduces a set of eight benchmark learning environments designed for evaluating RL algorithms in robot soccer. These environments are categorized as:

  • IEEE Very Small Size League (VSSS) Environments:
    • Single-Agent VSSS
    • Multi-Agent VSSS
  • Small Size League (SSL) Environments:
    • GoToBall (a basic skill)
    • Four environments based on RoboCup's 2021 hardware challenges:
  • Static Defenders * Contested Possession * Dribbling * Pass Endurance (single and multi-agent)

These environments are designed to be reproducible and cover a range of single-agent and multi-agent tasks, from basic skills to more complex competitive scenarios.

Key Insights and Ideas

  • Bridging the Sim-to-Real Gap: By providing a robust simulator and standardized environments, rSoccer aims to make it easier to develop RL policies that can potentially transfer to real robots, a critical step for robotics research.
  • Reproducibility is Key: The emphasis on an open-source framework and benchmark environments directly addresses the reproducibility crisis in RL research, allowing other teams to replicate results and build upon existing work.
  • Task Decomposition for RL: The framework breaks down complex robot soccer into manageable skills and challenges, allowing researchers to focus on specific aspects of RL for robotics.
  • Limitations of Current RL in Cooperation: The results highlight that even advanced RL algorithms struggle with true multi-agent cooperation, often leading to agents blocking each other rather than collaborating effectively.

Key Results

The paper evaluates two state-of-the-art deep RL algorithms, DDPG and SAC, across the proposed benchmark environments.

  • Performance: Both DDPG and SAC showed promising results in many environments, achieving performance that surpasses traditional handcrafted methods.
  • Sample Efficiency: DDPG was found to be more sample-efficient in most tasks, which is an interesting finding given SAC's typical superiority in continuous control. This is attributed to DDPG's Ornstein-Uhlenbeck process for exploration.
  • Multi-Agent Challenges: In multi-agent environments (IEEE VSSS and Pass Endurance), the results were worse than their single-agent counterparts. This suggests that the agents did not learn to collaborate effectively, with visual inspection revealing agents blocking each other.
  • Algorithm Limitations: Proximal Policy Optimization (PPO) was tested but was unable to learn even in the easiest environments, indicating potential issues with its application or parameter tuning for these specific tasks.
  • Benchmark Difficulty: Certain benchmarks, particularly the multi-agent environments and Static Defenders, proved to be too difficult for the current methods, highlighting areas for future research.

Limitations and Future Work

  • Multi-Agent Cooperation: A significant limitation is the difficulty in achieving effective collaboration in multi-agent scenarios. The authors suggest that specialized multi-agent algorithms like MADDPG could improve results.
  • PPO Performance: Understanding why PPO performed poorly is a key area for future investigation.
  • Benchmark Difficulty: The Static Defenders environment and multi-agent scenarios present significant challenges, indicating that the reward functions or observation spaces might need further refinement.
  • Sim-to-Real Transfer: While the framework facilitates sim-to-real research, direct evaluation on real robots is not detailed in this paper, though it's mentioned as a possibility.
  • Future Work: The authors suggest exploring more advanced multi-agent RL algorithms, investigating the poor performance of PPO, and potentially extending the framework to interface with real robots. They also aim to encourage more teams to adopt end-to-end RL approaches in robot soccer competitions.

Summary updated

BibTeX
@inproceedings{martins2021rsoccer,
  author = {Felipe B. Martins and Mateus G. Machado and Hansenclever F. Bassani and Pedro H. M. Braga and Edna Barros},
  title = {{rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer}},
  booktitle = {Robocup},
  year = {2021},
  doi = {10.1007/978-3-030-98682-7\_14},
}
Figure