Kick-motion Training with DQN in AI Soccer Environment

Bumgeun Park, Jihui Lee, Taeyoung Kim, Dongsoo Har

TL;DR

Researchers from KAIST have demonstrated that by simply switching from an absolute coordinate system to a robot-centric relative coordinate system, they can reduce the state space dimensionality for AI soccer robots by 60%. This approach effectively combats the "Curse of Dimensionality," allowing a Deep Q-Network (DQN) to learn complex kick-motions significantly faster and achieve higher performance than traditional methods.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), an agent learns by interacting with its environment. However, as the number of variables describing that environment (the "state") increases, the amount of data needed to train the agent grows exponentially. This is known as the Curse of Dimensionality (COD).

In the context of AI soccer, a robot using an Absolute Coordinate System (ACS) needs to track its own $(x, y)$ position on the field, its heading, the ball's $(x, y)$ position, and various velocities. For a specific task like kicking a ball, much of this global information is redundant. If the state space is too large and training data is sparse, the RL model often overfits or fails to converge, leading to poor on-field performance.

Methods

The authors propose using a Relative Coordinate System (RCS) to define the agent's state.

  1. Coordinate Transformation: Using a Coordinate Transformation Matrix (CTM), the authors translate the global field coordinates into a frame of reference where the robot is always at the origin $(0,0)$ and aligned with the x-axis.
  2. Dimensionality Reduction: In the ACS, the state vector requires 10 variables. By switching to RCS, several variables become constant (e.g., the robot's own position is always zero). This reduces the active state vector to just 6 variables: the ball's relative position $(x, y)$, the robot's forward velocity, the ball's relative velocities, and the ball's movement angle.
  3. Deep Q-Network (DQN): The researchers applied a standard DQN architecture to train the robot on this reduced state space.
  4. Reward Function: The agent is incentivized through a multi-part reward:
  5. r_contact: Reward for making contact with the ball.
  6. r_theta: Reward based on the angle of contact relative to the target.
  7. r_velocity: Reward based on the speed of the robot at the moment of contact.

Datasets

The study does not use a static dataset but rather a dynamic simulation environment:

  • Webots Physics Simulator: A professional mobile robot simulation software.
  • AI World Cup Environment: The specific simulation setup used for international AI soccer competitions, featuring two teams of five robots.

Key Insights and Ideas

  • Contextual Relevance: The "aha moment" is the realization that a robot doesn't need to know where it is on the field to perform a successful kick; it only needs to know where the ball is relative to itself.
  • Efficiency via Geometry: By using a CTM to shift the frame of reference, the researchers achieved a 60% reduction in state dimensionality without losing any information critical to the task.
  • Generalization: A major benefit of RCS is that once a robot learns to kick at one location on the field, that knowledge is immediately applicable anywhere else on the field because the relative geometry remains the same. In ACS, the robot would have to "re-learn" the motion for different absolute coordinates.

Key Results

The authors compared the RCS-based model against models with "dummy variables" to simulate the higher dimensionality of an ACS-based approach:

  • Faster Convergence: The RCS-based agent reached a total accumulated reward of -4 by the 1500th episode—roughly 500 episodes faster than the ACS-equivalent model.
  • Higher Peak Performance: The RCS method achieved a peak reward value of 6.8, outperforming the higher-dimensional models which struggled to reach the same level of proficiency.
  • Stability: The learning curve for the RCS model was more stable, whereas models with more dummy variables showed slower progress and more performance degradation due to the COD.

Limitations and Future Work

  • Task Specificity: This paper focuses exclusively on the "kick-motion." AI soccer involves many other sub-tasks (positioning, blocking, heading) that may require different state configurations.
  • Single Agent Focus: While conducted in a multi-agent environment, the paper focuses on training a single robot's motion. Future work could explore how RCS affects the coordination between multiple robots.
  • Sim-to-Real Gap: The study is entirely simulation-based. The authors do not discuss the challenges of implementing this coordinate transformation on physical hardware where sensor noise might affect the accuracy of the relative state.

Summary updated

Abstract

This paper presents a technique to train a robot to perform kick-motion in AI soccer by using reinforcement learning (RL). In RL, an agent interacts with an environment and learns to choose an action in a state at each step. When training RL algorithms, a problem called the curse of dimensionality (COD) can occur if the dimension of the state is high and the number of training data is low. The COD often causes degraded performance of RL models. In the situation of the robot kicking the ball, as the ball approaches the robot, the robot chooses the action based on the information obtained from the soccer field. In order not to suffer COD, the training data, which are experiences in the case of RL, should be collected evenly from all areas of the soccer field over (theoretically infinite) time. In this paper, we attempt to use the relative coordinate system (RCS) as the state for training kick-motion of robot agent, instead of using the absolute coordinate system (ACS). Using the RCS eliminates the necessity for the agent to know all the (state) information of entire soccer field and reduces the dimension of the state that the agent needs to know to perform kick-motion, and consequently alleviates COD. The training based on the RCS is performed with the widely used Deep Q-network (DQN) and tested in the AI Soccer environment implemented with Webots simulation software.
BibTeX
@inproceedings{park2023kick,
  author = {Bumgeun Park and Jihui Lee and Taeyoung Kim and Dongsoo Har},
  title = {{Kick-motion Training with DQN in AI Soccer Environment}},
  booktitle = {ICAIIC},
  year = {2023},
  url = {https://openreview.net/forum?id=KG2UfsTm6d},
}

Loading related content...

Figure