Unsupervised Discovery of 3D Physical Objects

Yilun Du, Kevin A. Smith, Tomer Ullman, Joshua B. Tenenbaum, Jiajun WuICLR 2021

TL;DR

This paper introduces POD-Net, a novel unsupervised method for discovering 3D physical objects from video. By integrating principles of physics and developmental psychology, POD-Net learns to segment and infer the 3D properties of objects, even when they are occluded, outperforming existing methods in both segmentation and physical reasoning tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of unsupervised physical object discovery from video. Traditional approaches often rely on 2D appearance cues for segmentation, which struggle with occlusions and inferring 3D properties. This limitation hinders a deeper understanding of scenes and the ability to reason about physical interactions. The authors aim to bridge this gap by leveraging the power of physics and motion cues, inspired by how infants learn to perceive and understand their world by observing object interactions and predictable motion patterns. This allows for a more robust disentanglement of object geometry and position from raw video data.

Methods

POD-Net is a self-supervised model composed of four key modules:

  • Inference Model: This module uses an attention network to sequentially infer a set of candidate object masks and their corresponding latent representations from an image. To handle objects of varying scales, it employs a sub-patch decomposition strategy, processing overlapping image patches to capture both coarse and fine details.
  • Backprojection Model: This module maps the inferred 2D segmentation masks to 3D primitive cuboids, representing objects with their translation, size, and rotation. This is achieved through a differentiable neural network, pre-trained to approximate the 2D-to-3D projection.
  • Dynamics Model: This module learns to predict the future states (position, size, rotation) of the 3D physical objects based on their current states. It uses a first-order approximation of velocity and angular velocity, enforcing that objects maintain their motion and shape over time.
  • Image Generative Model: This module, a VAE, decodes the latent representations and masks to reconstruct the original 2D image. This ensures that the learned object representations are visually consistent with the input video.

The overall system is trained by maximizing the joint likelihood of the physical dynamics and image generative models, with a focus on enforcing physical consistency.

Datasets

  • Synthetic Dataset: A dataset generated from ShapeNet objects, featuring various moving objects and occluders. Videos are 100 frames long, with objects moving in straight lines, back and forth, or rotating, but without collisions.
  • Real-world Dataset: A dataset of real block towers, with videos that may or may not show falling blocks. Each frame contains 2-4 blocks of different colors.

Key Insights and Ideas

  • Physics as a Constraint for Segmentation: The paper demonstrates that incorporating physical principles (like object permanence and smooth motion) significantly improves object segmentation, especially in challenging scenarios like occlusions. This is a departure from purely appearance-based methods.
  • Developmental Psychology Inspiration: Drawing inspiration from infant perception, the model leverages both motion cues and a foveation-like sub-patch processing approach to handle objects of different scales and complexities.
  • 3D Object Discovery from Unsupervised Video: The core contribution is the ability to discover not just 2D masks but also 3D geometric properties of objects from raw, unlabeled video, a significant step towards more human-like scene understanding.
  • Generalization to Novel Objects and Colors: POD-Net shows promising generalization capabilities, effectively segmenting novel objects and colors, highlighting its robustness beyond the training distribution.

Key Results

  • Segmentation Performance: POD-Net significantly outperforms baselines like MONet, OP3, and Normalized Cuts in terms of segmentation IoU and detection rates, particularly in scenarios with occlusions. For instance, on the ADEPT dataset with occluded objects, POD-Net (multi-scale, physics) achieved a detection rate of 0.734 and an average IoU of 0.701.
  • 3D Inference: The model achieves high 3D IoU and recall, demonstrating its ability to infer accurate 3D bounding boxes. A high correlation (r=0.615) was found between predicted and ground truth object translations.
  • Physical Reasoning: POD-Net's discovered object representations enable it to perform physical violation detection tasks, achieving competitive results on the "Block" task (relative accuracy of 0.622) and outperforming several baselines on the "Overturn (Long)" task (0.77).
  • Generalization: POD-Net shows improved performance on novel objects and colors compared to its performance on the training set, with physical consistency further boosting these results.

Limitations and Future Work

  • Spurious Detections: The paper acknowledges that over long videos, the model may discover several spurious extraneous objects, requiring hyperparameter tuning for specific tasks.
  • Cubeness Correlation: While POD-Net performs well on non-cuboid shapes, there's a relatively little correlation between well-segmented objects and "cubeness," suggesting room for improvement in handling more complex 3D shapes.
  • Future Work: The authors suggest further exploration in integrating more flexible representations of physical dynamics and continuing to advance robust object discovery through self-supervision. They believe this direction is promising for a richer understanding of the physical world.

Summary updated

Abstract

We study the problem of unsupervised physical object discovery. Unlike existing frameworks that aim to learn to decompose scenes into 2D segments purely based on each object's appearance, we explore how physics, especially object interactions, facilitates learning to disentangle and segment instances from raw videos, and to infer the 3D geometry and position of each object, all without supervision. Drawing inspiration from developmental psychology, our Physical Object Discovery Network (POD-Net) uses both multi-scale pixel cues and physical motion cues to accurately segment observable and partially occluded objects of varying sizes, and infer properties of those objects. Our model reliably segments objects on both synthetic and real scenes. The discovered object properties can also be used to reason about physical events.
BibTeX
@inproceedings{du2021unsupervised,
  author = {Yilun Du and Kevin A. Smith and Tomer Ullman and Joshua B. Tenenbaum and Jiajun Wu},
  title = {{Unsupervised Discovery of 3D Physical Objects}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}

Loading related content...

Figure