Inferring the Material Properties of Granular Media for Robotic Tasks

Carolyn Matl, Yashraj S. Narang, Ruzena Bajcsy, Fabio Ramos, Dieter Fox

TL;DR

This paper presents a framework that automatically calibrates physics simulators for granular materials (like grains or pellets) using only simple depth images of piles. By leveraging likelihood-free Bayesian inference, the system "guesses" hidden particle-scale properties—such as friction and restitution—allowing a robot to accurately predict how these materials will behave in complex manipulation tasks.

What Problem Does This Paper Address?

Granular materials are a nightmare for robotics. They don't behave strictly like solids or liquids, and their macroscale behavior (how a pile forms) is dictated by microscale properties (the friction between two tiny seeds) that are nearly impossible to measure directly without specialized lab equipment.

Currently, researchers using Discrete Element Method (DEM) simulators have to manually "tune" these parameters through tedious trial and error until the simulation looks "right." This paper asks: Can we automate this? Specifically, can a robot look at a pile of couscous, infer the underlying physics, and then use that knowledge to perform a task like pouring the couscous into a bowl without spilling?

Methods

The authors combine three high-tech components into a single pipeline:

  1. Simulation: They use NVIDIA’s Isaac Simulator, which utilizes a GPU-optimized solver to handle thousands of spherical particles simultaneously in real-time.
  2. Inference (BayesSim): This is the "brain" of the operation. Since simulators are complex "black boxes" where you can't easily calculate a mathematical likelihood, they use likelihood-free Bayesian inference. They run thousands of simulations with random parameters to learn a mapping between "pile shapes" and "physics parameters."
  3. Summary Statistics: Instead of comparing raw images (which is noisy and difficult), they extract 16 key statistics from depth images, such as the maximum height of the pile, the average radial distance of grains from the center, and how well the pile fits a chi-distribution.

The workflow is elegant: The system simulates 1,000 different piles, learns the relationship between the shapes and the physics, and then takes a single real-world depth image of a grain pile to estimate the most likely friction and restitution coefficients.

Datasets

  • Synthetic Data: 1,000 simulated grain formations used to train the BayesSim model.
  • Real-World Materials: The authors tested the system on Israeli couscous (roughly spherical) and barley (ellipsoidal).
  • Hardware: Observations were made using an off-the-shelf Intel RealSense D435 depth camera, and robotic tasks were performed by an ABB YuMi dual-arm industrial robot.

Key Insights and Ideas

  • Macro-to-Micro Inference: The most powerful idea here is that the "macro" shape of a pile (the angle of repose, the spread) contains enough information to back-calculate "micro" properties like rolling friction. You don't need a microscope; you just need a good statistical model of the pile.
  • The "Good Enough" Sphere: Even though barley is ellipsoidal and the simulator only uses spheres, the inference engine compensates. It finds "effective" friction values for spheres that mimic the behavior of the non-spherical real-world grains.
  • Likelihood-Free Flexibility: By using BayesSim, the framework is simulator-agnostic. You could swap NVIDIA Isaac for another engine, and the calibration process remains the same.

Key Results

  • High Precision: In "sim-to-sim" tests, the system could estimate sliding friction with an error of only 0.0062 after seeing 1,000 samples.
  • Generalization: Once calibrated on a simple 12cm pour, the simulator accurately predicted the shape of piles poured from different heights (2cm to 10cm).
  • Robotic Success:
    • The robot successfully created a complex "target" pattern of three concentric rings by inferring the exact heights needed to pour the grains.
  • The calibrated simulator accurately predicted the number of grains that would spill out of a bowl during a pouring task, matching real-world experimental results with "surprising accuracy."

Limitations and Future Work

  • Simplified Geometry: The current model assumes all grains are identical spheres. While the system compensates for this via friction values, it may struggle with highly irregular shapes (like long rice or flakes).
  • Static vs. Dynamic: The inference currently relies on "steady-state" images (after the pile has stopped moving). The authors suggest that using video sequences to capture the flow of grains could provide even richer data for inference.
  • Missing Physics: The model does not yet account for cohesion (sticky materials), moisture, or air drag (which affects very fine powders). Adding these would expand the system's utility to a wider range of industrial and culinary materials.

Summary updated

Abstract

Granular media (e.g., cereal grains, plastic resin pellets, and pills) are ubiquitous in robotics-integrated industries, such as agriculture, manufacturing, and pharmaceutical development. This prevalence mandates the accurate and efficient simulation of these materials. This work presents a software and hardware framework that automatically calibrates a fast physics simulator to accurately simulate granular materials by inferring material properties from real-world depth images of granular formations (i.e., piles and rings). Specifically, coefficients of sliding friction, rolling friction, and restitution of grains are estimated from summary statistics of grain formations using likelihood-free Bayesian inference. The calibrated simulator accurately predicts unseen granular formations in both simulation and experiment; furthermore, simulator predictions are shown to generalize to more complex tasks, including using a robot to pour grains into a bowl, as well as to create a desired pattern of piles and rings.
BibTeX
@inproceedings{matl2020inferring,
  author = {Carolyn Matl and Yashraj S. Narang and Ruzena Bajcsy and Fabio Ramos and Dieter Fox},
  title = {{Inferring the Material Properties of Granular Media for Robotic Tasks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=m09H2MtCAk},
}

Loading related content...

Figure