Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

Senthil Purushwalkam, Abhinav Gupta, Danny M. Kaufman, Bryan C. Russell

Introduction

The "Bounce and Learn" framework enables a system to predict the post-bounce trajectory of an object and infer hidden physical properties—like bounciness (restitution) and surface normals—from a single still image. By training on a massive new dataset of over 5,000 real-world ball bounces, the model learns to account for complex, non-rigid interactions (like sofas and beds) that traditional Newtonian physics simulators often fail to capture.

The Challenge of Real-World Physics

Predicting how an object will move after hitting a surface is a fundamental skill for humans, yet it remains incredibly difficult for AI. Traditional approaches usually fall into two camps:

  1. Purely Analytical: Using Newtonian physics equations. These work well for rigid billiard balls on slate tables but fail miserably when a ball hits a soft sofa or a cluttered desk, where surfaces deform and "effective" normals are hard to calculate.
  2. Purely Visual: Trying to guess physics from pixels. However, two surfaces can look identical but have vastly different physical responses.

The authors argue that to truly understand scene dynamics, an agent must combine visual recognition with physical interaction. The challenge lies in the "noisy" nature of the real world: high-speed collisions are hard to track, and real-world materials don't follow the clean, simplified rules of rigid-body simulators.

The "Bounce and Learn" Framework

The authors propose a dual-module architecture:

  • Physics Inference Module (PIM): This "engine" takes a pre-collision 3D trajectory (represented as a noisy point cloud) and a set of physical parameters to predict the post-collision trajectory. Instead of a standard regressor, PIM uses a non-parametric decoder. It maps the predicted physics to an embedding and performs a nearest-neighbor search in a database of 10,000 simulated trajectories, bypassing the difficulty of generating smooth 3D curves from scratch.
  • Visual Inference Module (VIM): This "eyes" module is a CNN that looks at a single RGB image and predicts the physical parameters—Coefficient of Restitution (COR) and surface normals—for every pixel in the scene.

The model is first bootstrapped using simple rigid-body simulations, then fine-tuned on real-world data. Crucially, PIM and VIM are trained jointly. This allows the PIM to learn "effective" physical parameters that accurately describe how a real-world, deformable surface behaves, even if those parameters don't perfectly align with textbook rigid-body physics.

The Bounce Dataset

To train this system, the authors introduce the Bounce Dataset:

  • Scale & Diversity: 5,172 stereo videos of a foam ball bouncing in everyday environments (offices, homes), capturing interactions with sofas, beds, chairs, walls, and floors.
  • Hardware: Captured at 100fps using a stereo camera to resolve the transient nature of collisions (which often last only 1/50th of a second).
  • Augmentation: Supplemented with 10,000 simulated trajectories from a physics engine for pre-training.

Key Insights and Results

The model uses visual cues to generalize, but relies on physical bounces to "calibrate" its understanding. It can even perform online learning, updating its estimate of a room's physics as it observes more bounces.

  • Trajectory Prediction: The model achieved a median error of 21.3 cm for the ball's position 0.1 seconds after the bounce, significantly outperforming baselines like parabola fitting (26.3 cm) and pure simulation-based models (40.1 cm).
  • Physical Inference: The VIM successfully learned to differentiate between soft surfaces and hard surfaces from images alone.
Inferred high restitution (COR 0.80) on a hard wooden cabinet surface.
Inferred low restitution (COR 0.08) on a soft fabric sofa, where traditional rigid-body physics often fails.

A key finding was that the model's improvement over traditional physics was most pronounced on low-restitution (soft) surfaces, where rigid-body assumptions are most frequently violated. Furthermore, in a robotics-style scenario, the model's accuracy in predicting normals and COR improved steadily as it observed more bounces in a new, unseen room.

Limitations and Future Work

While groundbreaking, the current model has limitations. It is trained using only one type of probe object (a spherical foam ball) and ignores friction and object spin, which significantly influence real-world trajectories. Additionally, it is deterministic and cannot capture the uncertainty of inherently stochastic surfaces (like a pile of clothes). Future directions include moving toward "learnable simulators" that can transfer recovered physical properties to 3D shapes for more complex AR/VR applications.

Summary updated

Abstract

We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajectories and infer two underlying physical properties that govern bouncing - restitution and effective collision normals. Our model, Bounce and Learn, comprises two modules -- a Physics Inference Module (PIM) and a Visual Inference Module (VIM). VIM learns to infer physical parameters for locations in a scene given a single still image, while PIM learns to model physical interactions for the prediction task given physical parameters and observed pre-collision 3D trajectories. To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices. Our proposed model learns from our collected dataset of real-world bounces and is bootstrapped with additional information from simple physics simulations. We show on our newly collected dataset that our model out-performs baselines, including trajectory fitting with Newtonian physics, in predicting post-bounce trajectories and inferring physical properties of a scene.
BibTeX
@inproceedings{purushwalkam2019bounce,
  author = {Senthil Purushwalkam and Abhinav Gupta and Danny M. Kaufman and Bryan C. Russell},
  title = {{Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=FxCtFZiWml5},
}
Figure