Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

Aly Magassouba, Komei Sugiura, Angelica Nakayama, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, Hisashi KawaiarXiv 2021 · arXiv

This paper, recently published by researchers from Japan's National Institute of Information and Communications Technology (NICT), Keio University, and Chubu University, introduces a novel approach to a common but complex robotics problem: how to place an object in a cluttered home environment without causing a mess.

TL;DR

The authors present PonNet, a multimodal neural network that predicts the likelihood of "damaging collisions" before a robot even begins its placing motion. By fusing RGB and depth data through a specialized attention mechanism, PonNet achieves nearly 91% accuracy and provides visual heatmaps explaining its risk assessment to the user.

What Problem Does This Paper Address?

Domestic service robots (DSRs) are increasingly expected to perform everyday manipulation tasks, like putting a glass on a crowded table. The challenge isn't just finding an empty spot; it's predicting the physical consequences of the action.

In a cluttered space, a robot might slightly nudge one object to make room for another. If that nudge is gentle, it’s fine. But if it causes a teapot to knock an hourglass off a shelf, it’s a "damaging collision." Traditional rule-based methods, which often rely on simple plane detection to find "free space," fail here because they lack an understanding of object physics and the chain reactions that occur in 3D space.

Figure 3: The PonNet architecture, featuring dual attention branches and a self-attention fusion layer.

Methods

The core contribution is PonNet, an architecture designed to handle the nuances of physical interaction through two main innovations:

  1. Multimodal Attention Branches: The network processes both RGB images and depth information (converted into surface normals). Unlike standard "black box" classifiers, PonNet uses an Attention Branch Network (ABN) for each modality. These branches generate attention maps that highlight the most salient features—like the edges of a precarious bottle—that contribute to collision risk.
  2. Self-Attention Fusion: Instead of simply concatenating the RGB and depth features, the authors use a self-attention layer to selectively weight the inputs. This is crucial because the two modalities are complementary: RGB is better at detecting transparent objects (like empty bottles), while depth is more robust for understanding 3D geometry.
Figure 3: The PonNet architecture, featuring dual attention branches and a self-attention fusion layer.

The final output is a probability score for a damaging collision (defined as a collision where relative speed exceeds 0.1 m/s) and a visualization of the risk areas.

Datasets

To train and validate PonNet, the authors created and released BILA-12K, a new large-scale dataset:

  • Scale: 12,000 photo-realistic RGB-D samples.
  • Environment: Procedurally generated home scenes using the Unity engine across 12 distinct locations.
  • Content: Features 28 types of obstacles and 15 types of graspable target objects.
  • Why it matters: Existing physical reasoning datasets like PHYRE (2D) or CLEVR (simple geometric shapes) lack the photo-realism and complex 3D physics required for real-world domestic robotics.

Key Insights and Ideas

  • Physics over Geometry: The model doesn't just look for "empty pixels." It learns to infer physical properties. For example, it can distinguish between a flat object that is safe to touch and a tall, narrow object that might tip over.
  • Explainability is a Feature: By overlaying attention maps on the input images, the robot can show a human user why it thinks a certain spot is risky. This builds trust and allows for better human-robot collaboration.
  • The Power of Fusion: The ablation studies confirmed that multimodal approaches consistently outperform monomodal ones. The self-attention mechanism specifically helped reduce false negatives (the most dangerous type of error) by 20% compared to using individual branches.

Key Results

  • Accuracy: PonNet achieved a best average test accuracy of 90.94%.
  • Baseline Comparison: It significantly outperformed a standard RGBD plane detection baseline, which only reached 82.5% accuracy.
  • Robustness: Qualitative analysis showed the model could generalize to the same obstacles in different orientations—recognizing, for instance, that a cup standing up is safer to nudge than a cup lying on its side that might roll into other objects.
Figure 6: Qualitative results showing true positives (risk detected) and true negatives (safe area detected) with corresponding attention heatmaps.

Limitations and Future Work

While PonNet is a significant step forward, the authors note a few areas for improvement:

  • Simplified Physics: The current simulation does not account for complex friction or object slippage, which are critical in real-world environments.
  • Rigid Bodies Only: The model does not yet handle deformable objects (like clothing or soft food).
  • Linguistic Integration: A promising future direction is combining this visual risk assessment with natural language processing, allowing the robot to explain its reasoning in plain English (e.g., "I'm not placing it there because that bottle looks unstable").

Summary updated

Abstract

Placing objects is a fundamental task for domestic service robots (DSRs). Thus, inferring the collision-risk before a placing motion is crucial for achieving the requested task. This problem is particularly challenging because it is necessary to predict what happens if an object is placed in a cluttered designated area. We show that a rule-based approach that uses plane detection, to detect free areas, performs poorly. To address this, we develop PonNet, which has multimodal attention branches and a self-attention mechanism to predict damaging collisions, based on RGBD images. Our method can visualize the risk of damaging collisions, which is convenient because it enables the user to understand the risk. For this purpose, we build and publish an original dataset that contains 12,000 photo-realistic images of specific placing areas, with daily life objects, in home environments. The experimental results show that our approach improves accuracy compared with the baseline methods.
BibTeX
@misc{magassouba2021predicting,
  author = {Aly Magassouba and Komei Sugiura and Angelica Nakayama and Tsubasa Hirakawa and Takayoshi Yamashita and Hironobu Fujiyoshi and Hisashi Kawai},
  title = {{Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations}},
  year = {2021},
  eprint = {2102.06507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.06507},
}

Loading related content...

Figure