An Analysis of Reinforcement Learning Applied to Coach Task in IEEE Very Small Size Soccer

Carlos H. C. Pena, Mateus G. Machado, Mariana S. Barros, José D. P. Silva, Lucas D. Maciel, Tsang Ing Ren, Edna Natividade da Silva Barros, Pedro H. M. Braga, Hansenclever F. Bassani

TL;DR

This paper introduces a Reinforcement Learning (RL) approach to automate the "coach" role in robot soccer, specifically for the IEEE Very Small Size Soccer (VSSS) league. By training an RL agent to dynamically select robot formations based on game state, the system achieves competitive performance against static opponents and even surpasses a human-designed deterministic coach.

What Problem Does This Paper Address?

In robot soccer, particularly in leagues like VSSS, teams of autonomous robots compete. A crucial aspect of a team's strategy is selecting the optimal formation and assigning roles (attacker, defender, goalkeeper) to each robot. Traditionally, this "coach" task is handled by a deterministic algorithm that relies on pre-defined rules and game state analysis. However, as games become more complex and competitive, a static or rule-based coaching strategy may not be sufficient. The authors identify the need for a more adaptive and intelligent coaching system that can learn optimal formations in real-time based on the evolving game situation, opponent strategies, and game dynamics. This paper aims to fill this gap by applying Reinforcement Learning to this "coach task."

Methods

The core of the proposed solution is a Reinforcement Learning agent trained using a Deep Q-Network (DDQN) and a Deep Deterministic Policy Gradient (DDPG) algorithm. The RL agent acts as the "coach," taking the game state as input and outputting a specific formation (a combination of roles for the three robots).

  • Environment: The authors leverage the FIRASim simulator, integrated with an OpenAI Gym-compatible environment, to create a realistic VSSS simulation. This setup allows for efficient training and evaluation of the RL agent.
  • State Space: The state representation includes continuous features like the ball's position, positions of both own and opponent robots, and their angles relative to the field center. To handle the temporal nature of coaching decisions, the state is represented by a stack of frames from the last few timesteps.
  • Action Space: The action space is discrete, representing 27 possible combinations of roles for the three robots. These combinations are further categorized into four basic strategies: Very Offensive, Offensive, Defensive, and Very Defensive.
  • RL Algorithms:
    • DDQN: A standard Deep Q-Network is used for its ability to handle discrete action spaces.
  • DDPG: This algorithm, designed for continuous action spaces, is adapted by discretizing its output into three ranges corresponding to Attacker, Defender, and Goalkeeper roles.
  • Reward Shaping: To address the sparsity of rewards (goals), the authors employ a reward shaping strategy that includes:
  • Positive reward for scoring a goal.
  • Negative reward for conceding a goal.
  • A penalty for committing fouls (e.g., having multiple defenders in the goal area with the ball).
  • The gradient of the ball's potential is also used to encourage more frequent rewards.

Datasets

The paper does not explicitly mention pre-existing datasets for training the coach. Instead, the training and evaluation are performed within a simulated environment (FIRASim) that generates game data dynamically. The authors train their RL agents against three different static opponent strategies: Balanced, Aggressive, and Heavily Aggressive. For evaluation, they also compare their trained agents against a specific team's binary code from the LARC 2019 competition.

Key Insights and Ideas

  • RL for Coaching is Viable: The paper demonstrates that Reinforcement Learning can effectively learn complex coaching strategies in a dynamic environment like robot soccer, outperforming traditional deterministic approaches.
  • Formation Choice is Crucial: The results highlight that the choice of formation significantly impacts game outcomes, and an adaptive strategy is superior to static ones.
  • DDPG's Interpretability: The authors note that the DDPG algorithm, despite requiring discretization, offers a degree of interpretability in its output, potentially aiding in understanding the learned strategy.
  • Offensive Strategies Tend to Perform Better: Surprisingly, the experiments suggest that balanced strategies are not as effective as offensive ones, possibly due to the complexity and effectiveness of the attacker's algorithm in the VSSS context.
  • Penalty Avoidance is Key: The inverse relationship between penalties and goal scores underscores the importance of penalizing fouls to encourage safer play.

Key Results

  • Training Performance: Both DDQN and DDPG agents showed increasing learning curves, indicating successful adaptation and strategy improvement over time.
  • Performance Against Static Opponents: The DDPG policy achieved a win/loss ratio of approximately 2.0 against a balanced opponent and showed promising results against aggressive opponents. The DDQN policy performed well only against the balanced team.
  • Comparison with LARC Team: The DDPG-trained coach was able to outperform a deterministic coach from a 5th-place LARC 2019 team, winning 14 out of 30 matches and achieving a better average score.
  • Action Distribution: In most experiments, the learned policies tended to favor formations with three attackers or a combination of attackers and one defender/goalkeeper.

Limitations and Future Work

  • Limited Role Complexity: The current system focuses on assigning roles (attacker, defender, goalkeeper) but assumes the underlying player-level control algorithms for these roles are already well-tuned.
  • Discrete Action Space Adaptation: The need to discretize the DDPG's continuous output is a workaround, and a more direct continuous control approach for formations might be beneficial.
  • Future Work: The authors propose:
  • Adding more diversity to the pre-defined formations.
  • Varying controller settings and maximum robot speeds.
  • Expanding the types of attacker behaviors.
  • Evaluating multi-agent RL methods for skills learning.
  • Developing a hierarchical approach where the coach controls team offensiveness and communicates with player agents.
  • Investigating sim-to-real transfer for deploying these learned strategies on physical robots.

Summary updated

Abstract

The IEEE Very Small Size Soccer (VSSS) is a robot soccer competition in which two teams of three small robots play against each other. Traditionally, a deterministic coach agent will choose the most suitable strategy and formation for each adversary's strategy. Therefore, the role of a coach is of great importance to the game. In this sense, this paper proposes an end-to-end approach for the coaching task based on Reinforcement Learning (RL). The proposed system processes the information during the simulated matches to learn an optimal policy that chooses the current formation, depending on the opponent and game conditions. We trained two RL policies against three different teams (balanced, offensive, and heavily offensive) in a simulated environment. Our results were assessed against one of the top teams of the VSSS league, showing promising results after achieving a win/loss ratio of approximately 2.0.
BibTeX
@inproceedings{pena2020analysis,
  author = {Carlos H. C. Pena and Mateus G. Machado and Mariana S. Barros and José D. P. Silva and Lucas D. Maciel and Tsang Ing Ren and Edna Natividade da Silva Barros and Pedro H. M. Braga and Hansenclever F. Bassani},
  title = {{An Analysis of Reinforcement Learning Applied to Coach Task in IEEE Very Small Size Soccer}},
  booktitle = {Lars/sbr/wre},
  year = {2020},
  doi = {10.1109/lars/sbr/wre51543.2020.9307069},
}
Figure