Interactive Perception of Granular Materials via Robotic Scooping and Force-Torque Sensing

Samuli Hynninen, Mohammad Amman, Meri Mäkelä, Anirvan Dutta, Ville KyrkiarXiv 2026 · arXiv

TL;DR

Researchers have developed Latent Filter++, a framework that allows robots to "feel" the physical properties of granular materials—like rice, sand, or pasta—simply by scooping them. By using a bio-inspired neural network that processes force-torque signals in real-time, the system can predict grain size and density, enabling the robot to adjust its movements for tasks like spill-free transportation.

What Problem Does This Paper Address?

Granular materials are a nightmare for roboticists. Unlike rigid objects, their behavior emerges from the complex interactions of thousands of individual particles. Factors like grain size, shape, and packing density drastically change how a material flows or resists a scoop.

Traditional approaches often rely on vision, which is frequently occluded during contact, or physics simulations that are notoriously difficult to calibrate to real-world materials. This paper tackles the challenge of interactive perception: inferring these hidden physical properties directly from the "kinesthetic" feedback (force and torque) experienced during a manipulation task. The goal is a system that works online, generalizes to new materials it hasn't seen before, and doesn't require hand-crafted signal processing.

Methods

The core of the approach is Latent Filter++, a deep state-space model designed for sequential inference.

  1. Bio-inspired Filter-Bank Encoder: Instead of using standard 1D convolutions, the authors designed an encoder inspired by human mechanoreceptors. It uses a learnable bank of frequency and temporal filters. This allows the model to explicitly decide which vibration frequencies (grain-scale contacts) and which time intervals (bulk interaction phases) are most informative.
  2. Hierarchical Latent Space: The model separates the state into two parts: $z_t$, which represents the immediate, observable interaction state, and $y_t$, which captures the underlying, slowly-varying physical properties of the material.
  3. Innovation Pathway: To help the model learn, the authors introduced an "innovation pathway" that feeds the discrepancy between predicted and actual forces back into the system. This helps the model "notice" when a material is behaving differently than expected, which is a strong signal for identifying its properties.
  4. Observation Masking: During training, the model is forced to predict missing chunks of sensor data, which pushes it to learn the underlying dynamics of the material rather than just memorizing the input signal.
Fig. 2: Proposed Latent Filter++ architecture for unsupervised deep state-space learning and inference.

Datasets

The researchers curated a diverse dataset of 26 everyday granular materials, categorized into five morphological classes:

  • Hard pieces: Big macaroni, peanuts, kidney beans.
  • Pebbles and seeds: Potting gravel, sunflower seeds, chickpeas.
  • Small grains/flakes: Barley, rice, oat flakes.
  • Fine granular: Couscous, breadcrumbs, sawdust.
  • Powders: Salt, sugar, wheat flour.

They collected 1,316 scooping recordings using a Franka Emika Panda robot equipped with a 6-axis force-torque sensor. Each recording includes 14 seconds of high-frequency (500 Hz) data.

Fig. 3: Selected granular materials (GM) used in the experiments. The color labels correspond to the latent-feature UMAP plots.

Key Insights and Ideas

  • Structure as Inductive Bias: By forcing the neural network to use frequency-temporal filter banks, the authors provided a "structured inductive bias." This makes the model much more robust to different observation window lengths compared to generic CNNs (see Figure 4).
  • Causality for Real-Time Use: Unlike previous "latent filter" models that required looking at a whole sequence at once, Latent Filter++ is strictly causal. It updates its belief about the material as the scoop progresses, making it suitable for real-time robot control.
  • Generalization via Latent Space: Because the model learns a continuous latent representation of physical properties rather than just discrete labels, it can estimate the properties of a completely new material (e.g., "leaving out" rice during training and testing on it later) much better than a standard supervised classifier.
  • Closing the Loop: The paper demonstrates that these inferred properties are actually useful. There is a strong correlation ($r_s = 0.74$) between the inferred grain size and the maximum acceleration a material can withstand before spilling out of a spoon.

Key Results

  • Encoder Performance: The proposed filter-bank encoder achieved 84.2% classification accuracy, significantly outperforming baselines like GmClass and SincNet, especially at short observation windows.
  • Property Estimation: Latent Filter++ reduced errors in grain size and bulk density estimation by 38% and 45%, respectively, compared to prior state-of-the-art latent filter methods.
  • Few-Shot Adaptation: The system can adapt to a brand-new material with as few as 1 to 5 scoops. By using these "anchor" interactions, the model can calibrate its property readouts without needing to retrain the entire neural network.
  • Online Convergence: As shown in Figure 6, the model's error (MAE) drops steadily as the scoop progresses, typically stabilizing well before the 14-second mark.
Temporal prediction of grain size and density shows the model's error decreasing as more interaction data is acquired.

Limitations and Future Work

  • Extrapolation Limits: While the model generalizes well within the range of materials it has seen, it struggles to extrapolate to materials that are vastly different from the training set (e.g., extremely large or strangely shaped particles).
  • Observability: Some properties, like the "angle of repose" (how steep a pile can be), were difficult to infer from scooping alone, suggesting that different types of interactions (like pouring or stirring) might be needed to "see" certain physics.
  • Autonomous Adaptation: Currently, the few-shot adaptation requires manual selection of "anchor" scoops. Future work aims to make the robot autonomously decide when it is uncertain and needs to perform "active perception" to learn more about a new material.

Summary updated

Abstract

Granular materials exhibit complex and highly variable interaction dynamics, making their reliable manipulation challenging without knowledge of their physical properties. We present an interactive perception framework that infers granular material properties directly from force-torque measurements acquired during scooping. The proposed Latent Filter++ combines a bio-inspired frequency-temporal filter-bank representation with a causal latent dynamical model, allowing the representation to evolve as interaction observations become available while avoiding hand-crafted signal preprocessing. The learned latent representation captures physically meaningful variation in granular materials and supports estimation of properties including grain size and bulk density. Evaluated on 26 diverse granular materials, the approach generalizes to materials excluded from training and enables few-shot adaptation of property-specific mappings using a small number of additional interaction samples. Finally, we demonstrate the relevance of the inferred properties to downstream manipulation by examining their correlation with the maximum acceleration the materials can withstand without spilling from the spoon during transport.
BibTeX
@misc{hynninen2026interactive,
  author = {Samuli Hynninen and Mohammad Amman and Meri Mäkelä and Anirvan Dutta and Ville Kyrki},
  title = {{Interactive Perception of Granular Materials via Robotic Scooping and Force-Torque Sensing}},
  year = {2026},
  eprint = {2611.15339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15339},
}
Figure