Recognizing object surface material from impact sounds for robot manipulation

Mariella Dimiccoli, Shubhan P. Patni, Matej Hoffmann, Francesc Moreno-NoguerIROS 2022 · DOI

Introduction

While computer vision has enabled significant advancements in robotic perception, it frequently encounters limitations when identifying the material composition of objects. For instance, a ceramic bowl and a plastic bowl may appear identical under specific lighting conditions or when viewed from certain angles, yet they require vastly different handling strategies during manipulation. Traditional alternatives, such as haptic sensing, often necessitate expensive and fragile specialized sensors.

This research explores an alternative "distal" sense: acoustics. When a robot interacts with an object—whether by poking, hitting, or dropping it—the resulting impact sound is fundamentally determined by the object's geometry and material properties. By enabling robots to "listen" to these interactions, we can bridge a critical gap in robotic "common sense," allowing manipulators to infer material properties through exploratory behaviors. This capability is essential for high-stakes tasks such as automated recycling, where material sorting is paramount, or the handling of fragile items.

Methodology

The proposed pipeline transforms raw acoustic data into material classifications through a multi-stage process:

  1. Interaction: The system initiates a physical interaction (e.g., hitting or poking) with an object to generate a characteristic sound.
  2. Preprocessing: To isolate the transient impact signal from ambient noise and robot motor sounds, a sliding window technique identifies the impact peak. The signal is then denoised using a wavelet-threshold multitaper spectra algorithm.
  3. Feature Extraction: The denoised audio is converted into a 128x128 grayscale spectrogram. This representation provides a visual "snapshot" of the sound's frequency components as they evolve over time.
  4. Classification: A modified ResNet34 architecture is employed for classification. Notably, this model is trained from scratch rather than using pre-trained ImageNet weights, as the transient, high-frequency nature of impact sounds differs significantly from the natural images typically used in computer vision.
Robotic experimental setup for capturing impact sounds during object interaction.

The YCB-Impact Sounds Dataset

A major contribution of this work is the introduction of the YCB-impact sounds dataset, designed to provide a standardized benchmark for acoustic robotic perception.

  • Contents: The dataset includes over 3,000 impact sounds and synchronized videos for 75 objects from the standard YCB (Yale-CMU-Berkeley) object set.
  • Recording Modes: To ensure robustness, sounds were recorded in three distinct modes: manual interactions (human-controlled) to capture high variability, teleoperated robot pokes (vertical), and autonomous robot pokes (horizontal) at varying speeds.
  • Categorization: Objects are labeled across 11 material classes, including steel, wood, glass, foam, and various types of plastic.
Example objects from the 'Paper' material category in the YCB-impact sounds dataset.

Results and Discussion

The experimental results highlight the potential of sound as a primary modality for material recognition.

  • Material Recognition: For unseen objects, the model achieved approximately 60% accuracy across 11 classes. While seemingly modest, this performance is nearly seven times better than random chance (9%), indicating a strong ability to generalize acoustic signatures to novel items.
  • Controlled Environment Performance: In a structured robotic environment using vertical poking, the classification accuracy reached 84.7%.
  • Object Identification: Beyond material classification, the model could identify specific individual objects from the 75-item set with 79% accuracy, demonstrating that impact sounds contain highly specific "fingerprints."
  • Generalization and Transferability: The study found that models trained on "noisy" manual data could be effectively fine-tuned for structured robotic environments, significantly outperforming models trained from scratch on limited robotic data.

However, certain materials remain challenging. Soft materials that damp sound, such as foam and rubber, produce weaker signals that are more easily obscured by background noise, leading to higher confusion rates between these categories.

Conclusion

This research demonstrates that robots can effectively identify object materials and identities by analyzing impact sounds. While challenges remain—such as decoupling the effects of geometry from material properties and handling highly damping materials—the integration of acoustic sensing offers a promising path toward more robust multi-modal robotic perception. Future work will focus on fusing auditory data with visual and tactile inputs to create a more comprehensive understanding of the physical world.

Summary updated

BibTeX
@inproceedings{dimiccoli2022recognizing,
  author = {Mariella Dimiccoli and Shubhan P. Patni and Matej Hoffmann and Francesc Moreno-Noguer},
  title = {{Recognizing object surface material from impact sounds for robot manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981578},
}
Figure