Abstract: Robotic interaction in fast-paced environments presents a substantial challenge, particularly in tasks requiring the prediction of dynamic, non-stationary objects for timely and accurate responses. An example of such a task is ping-pong, where the physical limitations of a robot may prevent it from reaching its goal in the time it takes the ball to cross the table. The scene of a ping-pong match contains rich visual information of a player's movement that can allow future game state prediction, with varying degrees of uncertainty. To this aim, we present a visual modeling, prediction, and control system to inform a ping-pong playing robot utilizing visual model uncertainty to allow earlier motion of the robot throughout the game. We present demonstrations and metrics in simulation to show the benefit of incorporating model uncertainty, the limitations of current standard model uncertainty estimators, and the need for more verifiable model uncertainty estimation.
Anticipatory Modeling in High-Speed Robotics
In fast-paced robotics, latency is a critical bottleneck. In ping-pong, by the time a ball is hit and its trajectory becomes physically clear, a robot arm may already be too far away to reach the strike point. To overcome this, robots must move beyond reactive behaviors and employ "anticipatory" models. These models analyze the opponent's body language and paddle movement to predict the ball's path before contact occurs.
However, early predictions are inherently unreliable. If a robot commits momentum toward an incorrect "guessed" location, it can be worse off than if it had remained stationary. Current frameworks often lack reliable confidence measures to inform the controller when to commit fully and when to "hedge its bets."
Methodology and Data Collection
The researchers developed a pipeline from data collection to an uncertainty-aware controller. The core model utilizes a Recurrent Neural Network (RNN) followed by a Multi-Layer Perceptron (MLP). The input is a 39-dimensional vector containing the opponent’s 8-joint skeleton, the paddle’s pose, and the ball’s 3D position. To simplify the task, the ball's path is parameterized as a piecewise linear trajectory on the $xy$-plane.
The system was trained on a custom dataset of over 2,000 hits collected from four hours of gameplay. Data was captured using four 100Hz RGB cameras and eight 120Hz OptiTrack cameras, with OpenPose used for human joint detection. Notably, the dataset exhibited a "right-hand bias" due to the natural habits of the human players, providing a unique challenge for the prediction model.
Evaluating Uncertainty Quantification (UQ)
The study tested several standard UQ methods to determine if they could reliably predict model error:
- Ensembles: Measuring variance between multiple trained models.
- Conformal Prediction: A statistical method providing a guaranteed confidence interval.
- k-Nearest Neighbors (k-NN): Utilizing historical errors from similar past inputs.
- Time-of-Inference: A simple heuristic where confidence is inversely proportional to the time remaining before the hit.
A key finding was that sophisticated methods like Ensembles and Conformal Prediction were largely ineffective for this dynamic task. Ensemble variance did not correlate well with actual error, and Conformal Prediction produced intervals so wide they offered no actionable guidance. Instead, the most reliable indicator of error was simply the time remaining before the hit; error scales almost linearly with "time-to-hit."
Experimental Results and Impact
The system was evaluated in a high-fidelity simulation using a KUKA LBR iiwa arm on a mobile Ridgeback base. By using the time-based confidence measure to scale the robot's velocity (via a factor $\alpha$), the robot could move slower when uncertain and at maximum speed when confident.
This uncertainty-aware approach significantly improved performance, particularly in addressing model bias. Because the dataset was skewed, the model often guessed "center-right" when unsure. A standard anticipatory model would rush to the right and miss balls hit to the left. The uncertainty-aware controller allowed the robot to stay more central when unsure, enabling it to recover and catch "left" hits once the trajectory became clearer.
Key Results: * Error Reduction: Median strike point error dropped from 26.27 cm (100ms before hit) to 16.65 cm (at hit). * Success Rate: The uncertainty-aware system achieved 151 successful hits, compared to 146 for basic anticipation and 144 for a reactive-only baseline.
Conclusion
While a simple time-based heuristic proved most effective in this study, it remains a global measure that does not account for specific shot difficulties. Future work will focus on developing "verifiable" uncertainty estimators capable of handling high-speed, non-stationary robotic tasks, potentially combining deep learning with physical laws through neuro-symbolic approaches.
Summary updated