Introduction
In high-speed sports like table tennis, reactive control—waiting to see where the ball goes before moving—is often too slow for robotic systems. Human experts excel because they anticipate: they read the opponent's body language and racket motion to predict the shot before it happens. While robotics researchers have built impressive table tennis systems, most rely on high-speed multi-camera setups in controlled labs. The "data bottleneck" has prevented the development of sophisticated anticipatory models because existing datasets are small and lack variety.
LATTE-MV introduces a scalable pipeline to reconstruct 3D table tennis gameplay from massive amounts of "in-the-wild" monocular YouTube videos. By training a transformer-based generative model on this data, the authors developed an uncertainty-aware controller that allows a robot to anticipate an opponent's intent, increasing its ball return rate from 49.9% to 59.0% in simulation.
Methodology
The authors propose a two-stage approach: Reconstruction and Anticipation.
3D Reconstruction from Monocular Video
To turn 2D broadcast footage into 3D data, the system uses a pipeline of specialized neural networks for entity tracking and global positioning.
- Entity Tracking: YOLOv8 segments the table and rackets, while HMR 2.0 estimates 3D human poses (SMPL meshes). TrackNetV3 handles the high-speed 2D ball tracking.
- Global Positioning: Since the camera is stationary but uncalibrated, the authors use the known geometry of a standard ITTF table and vanishing points to estimate camera intrinsics and extrinsics. This allows the system to map 2D coordinates into a consistent 3D world space.
- Physics-Informed Trajectory Fitting: They model the ball's 3D path using a projectile model that accounts for Stokes drag (air resistance), which is crucial for the light, high-drag nature of a ping-pong ball.
Anticipatory Prediction & Control
- The Model: A vanilla decoder-only transformer (3.2M parameters) learns the dynamics of the game. It treats the sequence of player poses and ball positions as "tokens," predicting future states auto-regressively.
- Uncertainty via Conformal Prediction: Instead of just predicting one point, the system uses Conformal Prediction to generate a "confidence region." This provides a statistical guarantee that the true ball position will be within that box a certain percentage of the time.
- The Controller: The robot (a simulated KUKA arm) uses these predictions to pre-position its racket in the predicted landing zone before the opponent even hits the ball.
The LATTE-MV Dataset
The authors introduced a massive new dataset derived from ~800 hours of raw online competition footage. This was distilled into 73,222 exchanges (~26.7 hours of pure gameplay). This is significantly larger than previous benchmarks, which often contained fewer than 15,000 segments. Unlike lab data, this features professional athletes in diverse competitive environments, capturing a wide variety of strategic styles.
Key Results and Insights
- Return Rate Improvement: In a MuJoCo simulation, the anticipatory policy achieved a 59.0% return rate, a nearly 10% absolute improvement over the non-anticipatory baseline (49.9%). An "Oracle" policy with perfect knowledge of the future achieved 64.5%, meaning the LATTE-MV model captures a substantial portion of the possible gains from anticipation.
- Scalability via Monocular Vision: The core insight is that while monocular reconstruction is noisier than multi-view lab setups, the sheer volume of data available on the internet compensates for the noise, allowing transformers to learn complex behavioral patterns.
- Strategic Bias: The data revealed that pro players have a bi-modal distribution in their shots, often hitting away from the center toward the corners. The transformer successfully captures this "strategic bias," allowing the robot to "buy time" by moving early.
- Efficiency: The model's inference time is <10ms on a standard consumer GPU (RTX 3090), making it viable for real-time robotic deployment.
Conclusion and Future Work
LATTE-MV demonstrates that large-scale, "in-the-wild" video data can effectively train robots for high-speed tasks. Future work aims to reconstruct exact racket orientation to better understand spin and to validate the uncertainty-aware controller on physical robotic platforms to test its resilience against real-world sensor noise and latency.
Summary updated