Introduction
The "Bounce and Learn" framework enables a system to predict the post-bounce trajectory of an object and infer hidden physical properties—like bounciness (restitution) and surface normals—from a single still image. By training on a massive new dataset of over 5,000 real-world ball bounces, the model learns to account for complex, non-rigid interactions (like sofas and beds) that traditional Newtonian physics simulators often fail to capture.
The Challenge of Real-World Physics
Predicting how an object will move after hitting a surface is a fundamental skill for humans, yet it remains incredibly difficult for AI. Traditional approaches usually fall into two camps:
- Purely Analytical: Using Newtonian physics equations. These work well for rigid billiard balls on slate tables but fail miserably when a ball hits a soft sofa or a cluttered desk, where surfaces deform and "effective" normals are hard to calculate.
- Purely Visual: Trying to guess physics from pixels. However, two surfaces can look identical but have vastly different physical responses.
The authors argue that to truly understand scene dynamics, an agent must combine visual recognition with physical interaction. The challenge lies in the "noisy" nature of the real world: high-speed collisions are hard to track, and real-world materials don't follow the clean, simplified rules of rigid-body simulators.
The "Bounce and Learn" Framework
The authors propose a dual-module architecture:
- Physics Inference Module (PIM): This "engine" takes a pre-collision 3D trajectory (represented as a noisy point cloud) and a set of physical parameters to predict the post-collision trajectory. Instead of a standard regressor, PIM uses a non-parametric decoder. It maps the predicted physics to an embedding and performs a nearest-neighbor search in a database of 10,000 simulated trajectories, bypassing the difficulty of generating smooth 3D curves from scratch.
- Visual Inference Module (VIM): This "eyes" module is a CNN that looks at a single RGB image and predicts the physical parameters—Coefficient of Restitution (COR) and surface normals—for every pixel in the scene.
The model is first bootstrapped using simple rigid-body simulations, then fine-tuned on real-world data. Crucially, PIM and VIM are trained jointly. This allows the PIM to learn "effective" physical parameters that accurately describe how a real-world, deformable surface behaves, even if those parameters don't perfectly align with textbook rigid-body physics.
The Bounce Dataset
To train this system, the authors introduce the Bounce Dataset:
- Scale & Diversity: 5,172 stereo videos of a foam ball bouncing in everyday environments (offices, homes), capturing interactions with sofas, beds, chairs, walls, and floors.
- Hardware: Captured at 100fps using a stereo camera to resolve the transient nature of collisions (which often last only 1/50th of a second).
- Augmentation: Supplemented with 10,000 simulated trajectories from a physics engine for pre-training.
Key Insights and Results
The model uses visual cues to generalize, but relies on physical bounces to "calibrate" its understanding. It can even perform online learning, updating its estimate of a room's physics as it observes more bounces.
- Trajectory Prediction: The model achieved a median error of 21.3 cm for the ball's position 0.1 seconds after the bounce, significantly outperforming baselines like parabola fitting (26.3 cm) and pure simulation-based models (40.1 cm).
- Physical Inference: The VIM successfully learned to differentiate between soft surfaces and hard surfaces from images alone.
A key finding was that the model's improvement over traditional physics was most pronounced on low-restitution (soft) surfaces, where rigid-body assumptions are most frequently violated. Furthermore, in a robotics-style scenario, the model's accuracy in predicting normals and COR improved steadily as it observed more bounces in a new, unseen room.
Limitations and Future Work
While groundbreaking, the current model has limitations. It is trained using only one type of probe object (a spherical foam ball) and ignores friction and object spin, which significantly influence real-world trajectories. Additionally, it is deterministic and cannot capture the uncertainty of inherently stochastic surfaces (like a pile of clothes). Future directions include moving toward "learnable simulators" that can transfer recovered physical properties to 3D shapes for more complex AR/VR applications.
Summary updated