Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

Senthil Purushwalkam, Abhinav Gupta, Danny M. Kaufman, Bryan C. Russell

Abstract

We introduce an approach to model surface properties governing bounces in everyday scenes. Our model, Bounce and Learn, learns end-to-end to predict post-bounce trajectories and infer underlying physical properties—restitution and effective collision normals—from sensor inputs. By leveraging a new dataset of 5,000 RGB-D videos of real-world bounces, our model outperforms traditional Newtonian physics baselines in predicting complex physical interactions.

Introduction: The Challenge of Real-World Physics

Humans intuitively grasp the physical properties of their environment, refining their expectations through interaction. For AI and robotics, however, predicting the outcome of high-speed physical collisions is notoriously difficult. Standard physics simulators typically assume objects are perfectly rigid, a simplification that fails when dealing with the soft, deformable materials found in homes and offices, such as sofas or pillows.

Predicting the post-bounce trajectory of an object on a soft, deformable surface like a sofa presents significant uncertainty for traditional physics engines.

Furthermore, extracting physical parameters like the Coefficient of Restitution (COR)—which dictates an object's "bounciness"—from visual data alone is challenging because visually similar surfaces may respond differently to impacts. The authors tackle this by building a system that visually estimates physical properties and forecasts high-speed collisions without relying on brittle, hand-coded engines.

The Bounce and Learn Framework

The proposed architecture consists of two integrated modules:

  • Visual Inference Module (VIM): A Convolutional Neural Network that takes a single RGB image and predicts a dense map of physical parameters. For every pixel, it estimates the COR and the effective collision normal.
  • Physics Inference Module (PIM): A neural physics engine that processes the pre-bounce 3D trajectory (represented as raw point clouds) and the VIM’s physical parameters to predict the post-bounce 3D trajectory.

The system is bootstrapped using simple rigid-body simulations and then fine-tuned on the Bounce Dataset, a first-of-its-kind collection of 5,172 high-framerate stereo RGB-D videos of a foam probe ball interacting with diverse surfaces in homes and offices.

Key Insights and Results

A major finding of this work is that real-world physics is inherently non-rigid. By fine-tuning on real data, the neural engine implicitly learns to model complex deformations—such as a ball sinking into a couch—that standard simulators miss. Additionally, using raw 3D point clouds via a PointNet-inspired encoder proved more robust than hand-crafted tracking, which often struggles with sensor noise during high-speed movement.

The Visual Inference Module (VIM) estimates physical properties like the Coefficient of Restitution (COR) from a single image; here, a wooden door is correctly identified as having high restitution (0.80).

The model achieved a median distance error of 21.3 cm for post-bounce predictions, significantly outperforming parabola-fitting baselines (26.3 cm). Qualitatively, the VIM successfully differentiates between soft objects (pillows, chair seats) and hard ones (tables, walls). The authors also demonstrated that "online learning"—observing multiple bounces in a new scene—allows the agent to progressively update and improve its visual map of the room's physical properties.

Conclusion

While the current model focuses on a single spherical probe and ignores factors like friction and spin, it represents a significant step toward robots that can visually perceive and reason about the physical dynamics of their environment. Future work will look toward detecting collisions "in the wild" and transferring these learned properties to advanced simulators for augmented reality and robotics.

Summary updated

Abstract

We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajectories and infer two underlying physical properties that govern bouncing - restitution and effective collision normals. Our model, Bounce and Learn, comprises two modules -- a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model physical interactions for the prediction task given physical parameters and observed pre-collision 3D trajectories. To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices. Our proposed model learns from our collected dataset of real-world bounces and is bootstrapped with additional information from simple physics simulations. We show on our newly collected dataset that our model out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.
BibTeX
@inproceedings{purushwalkam2019bounce,
  author = {Senthil Purushwalkam and Abhinav Gupta and Danny M. Kaufman and Bryan C. Russell},
  title = {{Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=FxCtFZiWml5},
}
Figure