Data-Efficient Online Learning of Ball Placement in Robot Table Tennis

Philip Tobuschat, Hao Ma, Dieter Büchler, Bernhard Schölkopf, Michael Muehlebach

Abstract

We present an implementation of an online optimization algorithm for hitting a predefined target when returning ping-pong balls with a table tennis robot. The online algorithm optimizes over so-called interception policies, which define the manner in which the robot arm intercepts the ball. In our case, these are composed of the state of the robot arm (position and velocity) at interception time. Gradient information is provided to the optimization algorithm via the mapping from the interception policy to the landing point of the ball on the table, which is approximated with a black-box and a grey-box approach. Our algorithm is applied to a robotic arm with four degrees of freedom that is driven by pneumatic artificial muscles. As a result, the robot arm is able to return the ball onto any predefined target on the table after about 2-5 iterations. We highlight the robustness of our approach by showing rapid convergence with both the black-box and the grey-box gradients. In addition, the small number of iterations required to reach close proximity to the target also underlines the sample efficiency.

Introduction

Achieving precise ball placement in robot table tennis is a significant challenge due to high-speed dynamics and complex hardware behavior. Traditional Reinforcement Learning (RL) approaches often suffer from extreme "data hunger," requiring millions of interactions. This is particularly problematic for robots driven by Pneumatic Artificial Muscles (PAMs), which exhibit non-linear, time-varying behavior that is difficult to model precisely.

The 4-DoF robot arm driven by pneumatic artificial muscles.

This work addresses these issues by proposing an Approximate Online Gradient Descent algorithm. The system aims to adapt to hardware noise and varying ball trajectories in real-time, achieving high accuracy with minimal data.

Methodology

The core of the approach is the optimization of an interception policy ($\phi$), which specifies the robot's joint positions and velocities at the moment of impact. Instead of learning full trajectories, the algorithm focuses on this critical state.

Gradient Approximation

To update the policy, the researchers approximate the gradient of the landing error. This is achieved through two distinct approaches:

  • Grey-Box Model: Utilizes first-principles physics, including a linear impact model and free-flight dynamics with air drag.
  • Black-Box Model: Employs a small multi-layer perceptron (MLP) trained on approximately 3,000 successful returns to predict landing points.

By decoupling the gradient into a model-based Jacobian and the measured landing error, the robot can iteratively refine its strike without a perfect global model of the PAM dynamics.

Results and Discussion

The system was evaluated on a physical 4-DoF robot arm. The results demonstrate exceptional data efficiency and robustness.

Rapid Convergence

The robot typically reaches the target vicinity within 2 to 5 iterations. This is orders of magnitude faster than standard RL methods. The distance error follows a theoretical convergence rate of $O(i^{-1/2})$, typical for online gradient descent.

Standard deviation of the landing error over 200 iterations for both black-box and grey-box models.

System Noise Floor

As shown in the performance plots, the standard deviation of the landing error converges to approximately 25 cm. This represents the physical "noise floor" of the system, caused by the inherent compliance of the PAMs and fluctuations in ball aerodynamics. Both the grey-box and black-box models successfully reached this limit, highlighting the algorithm's robustness to modeling errors.

Conclusion

This research demonstrates that online optimization of interception policies can overcome the data efficiency hurdles of traditional robotics learning. By using "good enough" gradient approximations, the robot achieves high-precision ball placement in just a few hits, even with complex, non-linear pneumatic actuators. Future work will look into incorporating ball spin to further reduce the error margin and extending the policy to include variable striking velocities.

Demonstration video: https://youtu.be/VC3KJoCss0k

Summary updated

Abstract

We present an implementation of an online optimization algorithm for hitting a predefined target when returning ping-pong balls with a table tennis robot. The online algorithm optimizes over so-called interception policies, which define the manner in which the robot arm intercepts the ball. In our case, these are composed of the state of the robot arm (position and velocity) at interception time. Gradient information is provided to the optimization algorithm via the mapping from the interception policy to the landing point of the ball on the table, which is approximated with a black-box and a grey-box approach. Our algorithm is applied to a robotic arm with four degrees of freedom that is driven by pneumatic artificial muscles. As a result, the robot arm is able to return the ball onto any predefined target on the table after about 2-5 iterations. We highlight the robustness of our approach by showing rapid convergence with both the black-box and the grey-box gradients. In addition, the small number of iterations required to reach close proximity to the target also underlines the sample efficiency. A demonstration video can be found here: https://youtu.be/VC3KJoCss0k.
BibTeX
@inproceedings{tobuschat2023data,
  author = {Philip Tobuschat and Hao Ma and Dieter Büchler and Bernhard Schölkopf and Michael Muehlebach},
  title = {{Data-Efficient Online Learning of Ball Placement in Robot Table Tennis}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342132},
}
Figure