Data-Efficient Online Learning of Ball Placement in Robot Table Tennis

Philip Tobuschat, Hao Ma, Dieter Büchler, Bernhard Schölkopf, Michael Muehlebach

This paper, presented by researchers from the Max Planck Institute for Intelligent Systems, tackles a classic challenge in robotics: making a robot play table tennis. Specifically, it focuses on ball placement—the ability to return a ball to a precise target on the table—using a highly non-linear, pneumatic-driven robot arm.

TL;DR

The researchers developed an online optimization algorithm that allows a table tennis robot to hit a specific target after only 2 to 5 practice hits. By combining a simple gradient descent approach with either a physics-based "grey-box" model or a neural network "black-box" model, the system achieves high accuracy and extreme data efficiency without needing thousands of trials or complex simulations.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has achieved impressive results in robotics, but it has a "data hunger" problem. Standard RL often requires thousands, if not millions, of interactions to learn a policy. In a high-speed, dynamic task like table tennis, this is impractical.

Furthermore, the robot used in this study is driven by Pneumatic Artificial Muscles (PAMs). While PAMs are "soft" and safe for human interaction, they are notoriously difficult to model because their behavior is non-linear and changes over time. Traditional controllers struggle with the inherent "noise" of the system and the varying trajectories of incoming balls. The authors sought a way to adapt to these changes online (during play) and with minimal data.

Methods

The core of the approach is Approximate Online Gradient Descent. Instead of learning a full movement trajectory from scratch, the authors optimize an interception policy ($\phi$). This policy defines the robot's state (position and velocity) at the exact moment it hits the ball.

The Secret Sauce: Gradient Approximation

To update the policy, the robot needs to know how a change in its swing affects where the ball lands. The authors split the gradient of the loss function into two parts:

  1. Part 1 (The Model): A Jacobian matrix that maps the interception state to the landing point.
  2. Part 2 (The Measurement): The actual observed error (how far the ball landed from the target).

The "Model" part doesn't have to be perfect. The authors tested two versions:

  • Grey-Box Model: A first-principles physics model including a linear impact model and free-flight dynamics with air drag.
  • Black-Box Model: A small, fully connected neural network (MLP) trained to predict landing points.

By using this approximation, the robot doesn't need to know the "true" underlying physics of the pneumatic muscles; it just needs a "good enough" direction to move its parameters.

Datasets

  • Training Data (Black-Box): To train the neural network for the black-box approach, the authors collected a dataset of approximately 3,000 successful returns.
  • Evaluation: The system was evaluated in real-time on a physical 4-degree-of-freedom (DoF) robot arm. No external benchmarks were used, as the focus was on the performance of this specific hardware setup.

Key Insights and Ideas

  • Robustness over Precision: One of the most surprising findings is that the algorithm is highly robust to modeling errors. Both the "grey-box" (which ignores complex spin) and the "black-box" (which is purely data-driven) resulted in similar convergence rates.
  • The Power of Prior Knowledge: By structuring the problem as an online optimization of an interception policy rather than an end-to-end RL problem, the authors drastically reduced the search space.
  • Handling System Noise: The robot has an inherent variance of about 25 cm (due to the "softness" of the PAMs and ball fluctuations). The algorithm successfully reaches this "noise floor" almost immediately, meaning it learns as much as is physically possible for the hardware.

Key Results

  • Rapid Convergence: The robot typically hits the target with an accuracy of 25 cm within just 2 to 5 iterations.
  • Data Efficiency: Compared to traditional RL methods that require thousands of iterations, this approach is orders of magnitude faster.
  • Real-time Performance: A full 200-iteration experiment takes roughly 30 minutes, during which the robot adapts to different targets and initializations without missing balls.
  • Quantitative Success: The distance error $\epsilon_i$ follows a convergence rate of roughly $O(i^{-1/2})$, aligning with theoretical expectations for online gradient descent.

Limitations and Future Work

  • Simplified Policy: To keep the optimization tractable, the authors fixed the robot's striking velocity ($\dot{\theta}_1$) to produce a "straight return." This means the robot isn't currently learning to vary the speed or "power" of the hit.
  • Ignoring Spin: The current models do not explicitly account for ball spin (the Magnus effect), treating it instead as process noise. Incorporating spin could further reduce the 25 cm error margin.
  • 2D vs. 3D: The current target is a 2D coordinate on the table surface. The authors note the approach could be extended to 3D targets (e.g., hitting a specific point in the air), but this hasn't been tested yet.

Summary updated

Abstract

We present an implementation of an online op-timization algorithm for hitting a predefined target when returning ping-pong balls with a table tennis robot. The online algorithm optimizes over so-called interception policies, which define the manner in which the robot arm intercepts the ball. In our case, these are composed of the state of the robot arm (position and velocity) at interception time. Gradient information is provided to the optimization algorithm via the mapping from the interception policy to the landing point of the ball on the table, which is approximated with a black-box and a grey-box approach. Our algorithm is applied to a robotic arm with four degrees of freedom that is driven by pneumatic artificial muscles. As a result, the robot arm is able to return the ball onto any predefined target on the table after about 2–5 iterations. We highlight the robustness of our approach by showing rapid convergence with both the black-box and the grey-box gradients. In addition, the small number of iterations required to reach close proximity to the target also underlines the sample efficiency. A demonstration video can be found here: https://youtu.be/VC3KJoCss0k.
BibTeX
@inproceedings{tobuschat2023data,
  author = {Philip Tobuschat and Hao Ma and Dieter Büchler and Bernhard Schölkopf and Michael Muehlebach},
  title = {{Data-Efficient Online Learning of Ball Placement in Robot Table Tennis}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342132},
}

Loading related content...

Figure