The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs

Christina Kassab, Matías Mattamala, Sacha Morin, Martin Büchner, Abhinav Valada, Liam Paull, Maurice F. FallonarXiv 2024 · DOI · arXiv

TL;DR

This paper tackles the computational cost of open-vocabulary 3D scene graph generation, a crucial capability for embodied AI. By systematically analyzing existing methods, they demonstrate that complex pre-processing and feature fusion strategies offer minimal accuracy gains while significantly increasing computation. They propose a streamlined approach that achieves comparable accuracy at a threefold reduction in computational cost, making real-time 3D scene understanding more feasible.

What Problem Does This Paper Address?

Embodied agents, like robots, need to understand their surroundings to navigate and interact effectively. A powerful way to represent a scene is through 3D scene graphs, which capture objects and their relationships. Open-vocabulary scene graphs are particularly important because they can recognize an unlimited number of object categories, moving beyond pre-defined lists. However, current state-of-the-art methods for generating these graphs are computationally very expensive, often relying on large models and complex pipelines that make them unsuitable for real-time applications. This paper investigates the critical design choices within these pipelines to identify inefficiencies and develop a simpler, more performant system.

Methods

The authors propose a general framework for open-vocabulary 3D scene graph methods and conduct three studies to analyze its components:

  • Image Pre-processing: They investigate the impact of scaling image crops and using segmentation masks (like SAM) on object classification accuracy.
  • Multi-View Feature Fusion: They examine how features extracted from different views of an object are combined, specifically focusing on averaging.
  • Multi-View Feature Selection: They explore strategies for selecting the most informative feature from multiple views, rather than fusing them.

Based on their findings, they develop a minimal system that uses geometry-based 3D segmentation, simple crop-based pre-processing, and entropy-based feature selection to generate per-object features.

Datasets

The study primarily uses two publicly available datasets:

  • ScanNet++: 10 representative scenes, including bedrooms, offices, apartments, and classrooms.
  • Replica: Scenes such as office0-office4 and room0-room2.

The authors also introduce an "extended prompt list" for evaluation, comprising 1687 labels from both datasets, which they argue is more realistic than using only ground-truth labels for a specific scene.

Key Insights and Ideas

  • Pre-processing is Overkill: Common image pre-processing techniques like multi-scale cropping and SAM masking provide minimal accuracy improvements but significantly increase computation time. A single crop scaled to 1.5x is often sufficient.
  • Averaging Hurts: Averaging features from multiple views of an object does not improve, and often degrades, classification accuracy. This is attributed to CLIP's sensitivity to viewpoint, where different views can yield very different predictions.
  • Selection is Key: Instead of fusing features, selecting the most informative feature from a single "optimal" view (determined by entropy) is a more effective strategy for improving accuracy without adding computational overhead.
  • Simplicity Wins: A minimal system that avoids complex pre-processing and feature fusion, relying on geometry for segmentation and entropy for feature selection, can achieve state-of-the-art performance with significantly reduced computational cost.

Key Results

  • The proposed minimal system achieves classification accuracy comparable to state-of-the-art methods like ConceptGraphs and HOV-SG.
  • The system achieves a threefold reduction in computation compared to existing methods.
  • Image pre-processing (cropping and masking) was found to add significant computational cost with little to no accuracy benefit. For instance, SAM and crop fusion increased computation time threefold compared to simple cropping.
  • Averaging multi-view features resulted in significantly lower accuracy compared to the "upper bound" (best possible accuracy from any single view).
  • Feature selection using Shannon entropy, particularly with a comprehensive prompt list, improved performance.

Limitations and Future Work

  • Segmentation Module: The authors acknowledge that their geometry-based region growing segmentation tends to over-segment objects. They suggest that incorporating newer class-free 3D segmentation methods like SAM3D could improve this.
  • Evaluation Metrics: The paper highlights challenges in evaluating open-vocabulary models due to CLIP's ability to assign semantically similar labels. They suggest the need for more flexible evaluation frameworks that capture the full versatility of language.
  • Real-time Performance: While their system is significantly faster, further optimization for real-time applications is still a goal. They mention that in an online setting, processing more views might be beneficial for optimal performance.
  • Edge Relationships: The current work focuses on generating graph nodes (object labels). Future work could explore building the graph and defining edge relationships.

Summary updated

Abstract

3D open-vocabulary scene graph methods are a promising map representation for embodied agents, however many current approaches are computationally expensive. In this paper, we reexamine the critical design choices established in previous works to optimize both efficiency and performance. We propose a general scene graph framework and conduct three studies that focus on image pre-processing, feature fusion, and feature selection. Our findings reveal that commonly used image pre-processing techniques provide minimal performance improvement while tripling computation (on a per object view basis). We also show that averaging feature labels across different views significantly degrades performance. We study alternative feature selection strategies that enhance performance without adding unnecessary computational costs. Based on our findings, we introduce a computationally balanced approach for 3D point cloud segmentation with per-object features. The approach matches state-of-the-art classification accuracy while achieving a threefold reduction in computation.
BibTeX
@misc{kassab2024bare,
  author = {Christina Kassab and Matías Mattamala and Sacha Morin and Martin Büchner and Abhinav Valada and Liam Paull and Maurice F. Fallon},
  title = {{The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs}},
  year = {2024},
  eprint = {2412.01539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01539},
}
Figure