Kick-motion Training with DQN in AI Soccer Environment

Bumgeun Park, Jihui Lee, Taeyoung Kim, Dongsoo Har

Abstract

This paper presents a technique to train a robot to perform kick-motion in AI soccer by using reinforcement learning (RL). In RL, an agent interacts with an environment and learns to choose an action in a state at each step. When training RL algorithms, a problem called the curse of dimensionality (COD) can occur if the dimension of the state is high and the number of training data is low. The COD often causes degraded performance of RL models. In the situation of the robot kicking the ball, as the ball approaches the robot, the robot chooses the action based on the information obtained from the soccer field. In order not to suffer COD, the training data, which are experiences in the case of RL, should be collected evenly from all areas of the soccer field over (theoretically infinite) time. In this paper, we attempt to use the relative coordinate system (RCS) as the state for training kick-motion of robot agent, instead of using the absolute coordinate system (ACS). Using the RCS eliminates the necessity for the agent to know all the (state) information of entire soccer field and reduces the dimension of the state that the agent needs to know to perform kick-motion, and consequently alleviates COD. The training based on the RCS is performed with the widely used Deep Q-network (DQN) and tested in the AI Soccer environment implemented with Webots simulation software.

Introduction

In Reinforcement Learning (RL), an agent learns by interacting with its environment. However, as the number of variables describing that environment (the "state") increases, the amount of data needed to train the agent grows exponentially. This is known as the Curse of Dimensionality (COD). In AI soccer, a robot using an Absolute Coordinate System (ACS) must track its global position, heading, and the ball's position. For a specific task like kicking, much of this global information is redundant. If the state space is too large and training data is sparse, the RL model often overfits or fails to converge.

Methodology

Relative Coordinate System (RCS)

The researchers propose using a Relative Coordinate System (RCS) to define the agent's state. Using a Coordinate Transformation Matrix (CTM), global field coordinates are translated into a frame of reference where the robot is always at the origin $(0,0)$ and aligned with the x-axis.

The Relative Coordinate System (RCS) centers the frame of reference on the robot, reducing the complexity of the state space.

In the ACS, the state vector requires 10 variables. By switching to RCS, several variables become constant (e.g., the robot's own position is always zero). This reduces the active state vector to just 6 variables: the ball's relative position $(x, y)$, the robot's forward velocity, the ball's relative velocities, and the ball's movement angle. This represents a 60% reduction in state dimensionality.

Training and Rewards

The training was conducted using a Deep Q-Network (DQN) within the Webots physics simulator. The agent is incentivized through a multi-part reward function:

  • Contact Reward ($r_{contact}$): Reward for making contact with the ball.
  • Angle Reward ($r_{\theta}$): Reward based on the angle of contact relative to the target.
  • Velocity Reward ($r_{velocity}$): Reward based on the speed of the robot at the moment of contact.

Results

The RCS-based model was compared against models with "dummy variables" to simulate the higher dimensionality typical of ACS-based approaches. The results demonstrate that lower dimensionality leads to significantly better learning outcomes.

Comparison of training performance across different state dimensions. The RCS model (+0 dummy variables) shows faster convergence and higher peak rewards.

Key findings include:

  • Faster Convergence: The RCS-based agent reached a total accumulated reward of -4 by the 1500th episode—roughly 500 episodes faster than higher-dimensional models.
  • Higher Peak Performance: The RCS method achieved a peak reward value of 6.8, outperforming models that struggled with the Curse of Dimensionality.
  • Generalization: A major benefit of RCS is that once a robot learns to kick at one location, that knowledge is immediately applicable anywhere else on the field because the relative geometry remains constant.

Conclusion

By simply switching from an absolute to a robot-centric relative coordinate system, the researchers effectively mitigated the Curse of Dimensionality in AI soccer training. This approach allows for more efficient data usage and more stable convergence in RL tasks. Future work may explore applying RCS to multi-agent coordination and addressing the sim-to-real gap regarding sensor noise in physical hardware.

Summary updated

Abstract

This paper presents a technique to train a robot to perform kick-motion in AI soccer by using reinforcement learning (RL). In RL, an agent interacts with an environment and learns to choose an action in a state at each step. When training RL algorithms, a problem called the curse of dimensionality (COD) can occur if the dimension of the state is high and the number of training data is low. The COD often causes degraded performance of RL models. In the situation of the robot kicking the ball, as the ball approaches the robot, the robot chooses the action based on the information obtained from the soccer field. In order not to suffer COD, the training data, which are experiences in the case of RL, should be collected evenly from all areas of the soccer field over (theoretically infinite) time. In this paper, we attempt to use the relative coordinate system (RCS) as the state for training kick-motion of robot agent, instead of using the absolute coordinate system (ACS). Using the RCS eliminates the necessity for the agent to know all the (state) information of entire soccer field and reduces the dimension of the state that the agent needs to know to perform kick-motion, and consequently alleviates COD. The training based on the RCS is performed with the widely used Deep Q-network (DQN) and tested in the AI Soccer environment implemented with Webots simulation software.
BibTeX
@inproceedings{park2023kick,
  author = {Bumgeun Park and Jihui Lee and Taeyoung Kim and Dongsoo Har},
  title = {{Kick-motion Training with DQN in AI Soccer Environment}},
  booktitle = {ICAIIC},
  year = {2023},
  url = {https://openreview.net/forum?id=KG2UfsTm6d},
}
Figure