Abstract
We introduce an approach to model surface properties governing bounces in everyday scenes. Our model, Bounce and Learn, learns end-to-end to predict post-bounce trajectories and infer underlying physical properties—restitution and effective collision normals—from sensor inputs. By leveraging a new dataset of 5,000 RGB-D videos of real-world bounces, our model outperforms traditional Newtonian physics baselines in predicting complex physical interactions.
Introduction: The Challenge of Real-World Physics
Humans intuitively grasp the physical properties of their environment, refining their expectations through interaction. For AI and robotics, however, predicting the outcome of high-speed physical collisions is notoriously difficult. Standard physics simulators typically assume objects are perfectly rigid, a simplification that fails when dealing with the soft, deformable materials found in homes and offices, such as sofas or pillows.
Furthermore, extracting physical parameters like the Coefficient of Restitution (COR)—which dictates an object's "bounciness"—from visual data alone is challenging because visually similar surfaces may respond differently to impacts. The authors tackle this by building a system that visually estimates physical properties and forecasts high-speed collisions without relying on brittle, hand-coded engines.
The Bounce and Learn Framework
The proposed architecture consists of two integrated modules:
- Visual Inference Module (VIM): A Convolutional Neural Network that takes a single RGB image and predicts a dense map of physical parameters. For every pixel, it estimates the COR and the effective collision normal.
- Physics Inference Module (PIM): A neural physics engine that processes the pre-bounce 3D trajectory (represented as raw point clouds) and the VIM’s physical parameters to predict the post-bounce 3D trajectory.
The system is bootstrapped using simple rigid-body simulations and then fine-tuned on the Bounce Dataset, a first-of-its-kind collection of 5,172 high-framerate stereo RGB-D videos of a foam probe ball interacting with diverse surfaces in homes and offices.
Key Insights and Results
A major finding of this work is that real-world physics is inherently non-rigid. By fine-tuning on real data, the neural engine implicitly learns to model complex deformations—such as a ball sinking into a couch—that standard simulators miss. Additionally, using raw 3D point clouds via a PointNet-inspired encoder proved more robust than hand-crafted tracking, which often struggles with sensor noise during high-speed movement.
The model achieved a median distance error of 21.3 cm for post-bounce predictions, significantly outperforming parabola-fitting baselines (26.3 cm). Qualitatively, the VIM successfully differentiates between soft objects (pillows, chair seats) and hard ones (tables, walls). The authors also demonstrated that "online learning"—observing multiple bounces in a new scene—allows the agent to progressively update and improve its visual map of the room's physical properties.
Conclusion
While the current model focuses on a single spherical probe and ignores factors like friction and spin, it represents a significant step toward robots that can visually perceive and reason about the physical dynamics of their environment. Future work will look toward detecting collisions "in the wild" and transferring these learned properties to advanced simulators for augmented reality and robotics.
Summary updated