Two-stage training algorithm for AI robot soccer

Taeyoung Kim, Luiz Felipe Vecchietti, Kyujin Choi, Sanem Sariel, Dongsoo Har

Abstract

Multi-agent reinforcement learning (MARL) in heterogeneous environments, such as robot soccer, faces significant challenges due to non-stationarity and the need for specialized role behaviors. This paper introduces the Two-stage Heterogeneous Centralized Training (TSHCT) algorithm. By decoupling training into role-specific optimization and team-level coordination, TSHCT enables agents to master individual skills while maintaining effective group strategy. Experimental results in a high-fidelity 5v5 simulation demonstrate that TSHCT significantly outperforms established MARL benchmarks like QMIX and VDN in both reward accumulation and win rates.

Introduction

Teaching a team of robots to play soccer is a classic challenge in artificial intelligence. It requires solving the "non-stationarity" problem inherent in multi-agent reinforcement learning: as one agent learns and changes its behavior, the environment effectively changes for all other agents. This complexity is heightened in heterogeneous teams where different robots (e.g., goalkeepers vs. forwards) require distinct skill sets and physical properties.

Existing frameworks often utilize Centralized Training with Decentralized Execution (CTDE). However, these methods frequently struggle to balance specialized role learning with team-wide coordination. Purely collective rewards can lead to a lack of individual specialization, while purely individual rewards can result in selfish behavior that undermines team performance.

Methods: Two-Stage Heterogeneous Centralized Training (TSHCT)

The proposed TSHCT algorithm addresses these challenges through a serial training structure executed at every timestep, dividing the learning process into two distinct phases.

Stage 1: Role-Specific Training

In the first stage, agents are grouped by role (e.g., Goalkeeper, Defenders, Forwards). Agents of the same type share a policy and are trained to maximize individual role rewards. These are "dense" rewards based on immediate feedback such as ball distance, velocity, and maintaining optimal positioning. This ensures that a goalkeeper learns defensive fundamentals while forwards focus on offensive pressure.

Stage 2: Team-Level Training

The second stage integrates these specialized behaviors into a cohesive team strategy. A "Team Mixing Network," implemented as a hypernetwork conditioned on the global state, aggregates the outputs from the role-specific networks. This stage focuses on maximizing the shared collective reward, which consists of "sparse" feedback such as scoring or conceding goals.

The TSHCT architecture showing the two-stage training process: role-specific optimization followed by team-level mixing.

To handle partial observability and fast-paced dynamics, the system utilizes Dueling Deep Recurrent Q-Networks (DRQN) with Gated Recurrent Units (GRUs), allowing robots to maintain a temporal memory of the field. Furthermore, self-play is employed to provide a curriculum of increasing difficulty by periodically updating the opponent to a previous version of the learning team.

Experimental Setup

The algorithm was evaluated in a high-fidelity Webots Robot Simulation environment configured for 5v5 matches. The team composition included one Goalkeeper (GK), two Defenders (D1, D2), and two Forwards (F1, F2). Each role featured unique physical specifications: the Goalkeeper was the heaviest (2.5kg) and slowest (1.8 m/s) for stability, while Forwards were the lightest (1.5kg) and fastest (2.55 m/s) for agility.

Key Results

TSHCT was benchmarked against three prominent MARL algorithms: COMA, VDN, and QMIX. The results demonstrated superior performance across multiple metrics:

  • Reward Accumulation: TSHCT consistently achieved higher total rewards and role-specific rewards compared to all baselines over 200,000 training episodes.
  • Score-Concede Rate: TSHCT achieved a score-concede rate of 79.01% against a COMA opponent, outperforming QMIX (67.3%) and VDN (64.2%).
  • Win Rates: In direct head-to-head competition, the TSHCT team maintained an 80% to 100% win rate against teams trained with baseline algorithms.
Comparative performance graphs showing TSHCT outperforming QMIX, VDN, and COMA in total and role-specific rewards.

Ablation studies further confirmed that the combination of the two-stage training approach with the Dueling DRQN architecture provided the most significant performance boost, improving the score-concede rate by up to 18% depending on the opponent.

Conclusion and Future Work

The TSHCT algorithm successfully bridges the gap between individual specialization and team coordination in heterogeneous multi-agent environments. By utilizing a two-stage reward structure, the system effectively solves the credit assignment problem in robot soccer.

Future research will focus on transitioning from discrete to continuous action spaces for smoother robotic control and exploring the integration of this two-stage approach with actor-critic policy-based algorithms. Additionally, addressing the "sim-to-real" gap remains a priority for deploying these strategies on physical hardware.

Summary updated

BibTeX
@article{kim2021two,
  author = {Taeyoung Kim and Luiz Felipe Vecchietti and Kyujin Choi and Sanem Sariel and Dongsoo Har},
  title = {{Two-stage training algorithm for AI robot soccer}},
  journal = {PeerJ Comput. Sci.},
  year = {2021},
  doi = {10.7717/peerj-cs.718},
}
Figure