GLAVNet: Global-Local Audio-Visual Cues for Fine-Grained Material Recognition

Fengmin Shi, Jie Guo, Haonan Zhang, Shan Yang, Xiying Wang, Yanwen GuoCVPR 2021 · OpenReview

Abstract

In this paper, we aim to recognize materials with combined use of auditory and visual perception. To this end, we construct a new dataset named GLAudio that consists of both the geometry of the object being struck and the sound captured from either modal sound synthesis (for virtual objects) or real measurements (for real objects). Besides global geometries, our dataset also takes local geometries around different hitpoints into consideration. This local information is less explored in existing datasets. We demonstrate that local geometry has a greater impact on the sound than the global geometry and offers more cues in material recognition. To extract features from different modalities and perform proper fusion, we propose a new deep neural network GLAVNet that comprises multiple branches and a well-designed fusion module. Once trained on GLAudio, our GLAVNet provides state-of-the-art performance on material identification and supports fine-grained material categorization.

Introduction

Identifying the material composition of objects is a fundamental challenge in computer vision, with significant implications for robotic manipulation and immersive AR/VR experiences. While visual cues are often the primary source of information, they can be deceptive; for instance, a plastic surface can be finished to mimic wood or metal perfectly. To overcome these limitations, researchers have increasingly turned to "impact sounds" as a complementary modality.

Striking different locations on an object, such as the handle (green) versus the body (orange), produces distinct acoustic signatures due to local geometric variations.

However, a critical physical reality often overlooked is that the sound an object makes depends heavily on the hitpoint—the specific location where it is struck. Striking the thin handle of a ceramic mug produces a vastly different acoustic profile than striking its thick base. GLAVNet addresses this by incorporating hitpoint-aware local geometry, moving beyond coarse material classification to achieve fine-grained identification (e.g., distinguishing between aluminum and tin).

The GLAVNet Framework

GLAVNet is a multi-branch convolutional neural network (CNN) designed to fuse three distinct types of information to predict material properties:

  1. Global Geometry Subnetwork: Utilizes a 3D CNN (VoxNet) to process a coarse voxelized representation of the entire object's shape.
  2. Local Geometry Subnetwork: Focuses on a high-resolution voxel grid centered specifically on the hitpoint. This captures the local thickness and curvature that primarily dictate vibration modes.
  3. Audio Subnetwork: Converts the impact sound into a Mel-spectrogram, which is then processed by a 2D CNN to extract deep acoustic features.

To integrate these modalities, the model employs Multi-modal Factorized Bilinear (MFB) pooling. Unlike simple concatenation, MFB pooling allows the model to learn complex, non-linear correlations between shape and sound, ensuring that visual cues do not overshadow subtle but critical acoustic information.

The GLAudio Dataset

To train and evaluate GLAVNet, the authors introduced GLAudio, a dataset specifically designed to capture the relationship between geometry and impact sound. It contains over 40,000 synthesized samples and real-world measurements across 17 fine-grained material categories, including 10 types of metal and 4 types of wood.

The GLAudio dataset pairs high-resolution local geometry voxels with their corresponding synthesized impact sound spectrograms.

For virtual objects, sounds were generated using modal sound synthesis, a physics-based approach that calculates vibrations based on material properties like density and Young’s modulus. Crucially, GLAudio explicitly records the hitpoint and the local geometry surrounding it, a feature missing from previous datasets like Sound-20K.

Key Insights and Results

Experiments conducted with GLAVNet yielded several important insights:

  • Local Geometry Dominance: The research proves that local geometry around the hitpoint is more influential on the resulting sound than the object's global shape. Striking a cup's body produces similar sounds regardless of whether the handle exists, but striking the handle itself creates a unique signature.
  • Fine-Grained Accuracy: GLAVNet significantly outperforms previous models in fine-grained tasks. On a 17-category dataset, GLAVNet achieved 75.3% accuracy, compared to 65.8% for the previous state-of-the-art (ISNN-AV).
  • Real-World Generalization: Despite being trained primarily on synthetic data, GLAVNet achieved 53.4% accuracy on real-world objects, nearly doubling the performance of baseline models (approx. 28%).
  • Ablation Studies: Removing the local geometry branch led to a significant performance drop, confirming that "where you hit" is vital for accurate material recognition.

Conclusion and Future Work

GLAVNet demonstrates that combining global and local geometric cues with audio significantly enhances material recognition. While the model shows strong performance, future work will focus on expanding the real-world dataset and incorporating a wider variety of sounds, such as scraping or sliding. Additionally, extending the framework to handle deformable or soft objects remains an open research challenge.

Summary updated

Abstract

In this paper, we aim to recognize materials with combined use of auditory and visual perception. To this end, we construct a new dataset named GLAudio that consists of both the geometry of the object being struck and the sound captured from either modal sound synthesis (for virtual objects) or real measurements (for real objects). Besides global geometries, our dataset also takes local geometries around different hitpoints into consideration. This local information is less explored in existing datasets. We demonstrate that local geometry has a greater impact on the sound than the global geometry and offers more cues in material recognition. To extract features from different modalities and perform proper fusion, we propose a new deep neural network GLAVNet that comprises multiple branches and a well-designed fusion module. Once trained on GLAudio, our GLAVNet provides state-of-the-art performance on material identification and supports fine-grained material categorization.
BibTeX
@inproceedings{shi2021glavnet,
  author = {Fengmin Shi and Jie Guo and Haonan Zhang and Shan Yang and Xiying Wang and Yanwen Guo},
  title = {{GLAVNet: Global-Local Audio-Visual Cues for Fine-Grained Material Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=m2ZAVSZhdPE},
}
Figure