Few-Shot Structure-Informed Machinery Part Segmentation with Foundation Models and Graph Neural Networks

Michael Schwingshackl, Fabio F. Oberweger, Markus MurschitzWACV 2025

TL;DR

This paper introduces a novel approach for segmenting parts of complex machinery using a few annotated examples. By combining powerful pre-trained foundation models (CLIPSeg, SAM, SuperPoint) with a custom graph neural network, the method achieves robust and efficient few-shot segmentation, even generalizing to real-world data after training on synthetic images.

What Problem Does This Paper Address?

Autonomous systems need to understand and interact with human-made structures, which often consist of multiple interconnected parts. Identifying these parts and their relationships is crucial for tasks like manipulation, navigation, and state assessment. However, training models for this task typically requires large amounts of precisely annotated data, which is expensive and time-consuming to acquire, especially for specialized machinery. This paper tackles the challenge of few-shot semantic segmentation for machinery parts, aiming for a system that can learn to segment new structures with minimal annotated examples and fast training times, making it adaptable for real-world autonomous applications.

Methods

The core of the proposed method lies in orchestrating several pre-trained foundation models with a novel graph neural network (GNN) architecture.

  • Foundation Models:
    • CLIPSeg: Used to generate initial likelihood maps for specific text prompts (e.g., "crane," "truck"). This leverages the text-image understanding capabilities of large language models.
  • SuperPoint: Detects interest points (keypoints) within an image, providing robust feature locations that are less sensitive to lighting and perspective changes.
  • Segment Anything Model (SAM): A powerful general-purpose segmentation model that requires prompts (like points or bounding boxes) to generate masks.
  • Graph Construction: Interest points from SuperPoint, enhanced with CLIPSeg likelihoods, form the nodes of a graph. Edges connect these nodes based on spatial proximity. This graph representation captures both visual features and spatial relationships between potential object parts.
  • Graph Neural Network (GNN): A custom, few-shot trainable GNN acts as the classifier. It takes the graph as input and assigns a class label (e.g., "cabin," "wheel") to each node. This is the only part of the system that is trained.
  • SAM Prompt Engineering: A novel process converts the GNN's node classifications into effective prompts for SAM. This involves grouping classified nodes, filtering outliers, and generating bounding boxes and point prompts for SAM to produce precise final segmentations.

The key novelty lies in the integration and orchestration of these models, particularly the GNN-based classification on a graph representation and the subsequent SAM prompt engineering.

Datasets

  • Synthetic Dataset: A custom synthetic dataset of a truck-mounted loading crane was generated using Blender. This dataset features extensive domain randomization (lighting, camera perspective, crane articulation) to create diverse renderings. It includes annotations at various granularity levels, from coarse (truck, crane arm) to fine-grained (22 individual parts).
  • DAVIS 2017 Dataset: Used to evaluate the system's adaptability and performance in semi-supervised video segmentation tasks, demonstrating its generalization beyond the specific machinery domain.

Key Insights and Ideas

  • Foundation Model Synergy: Combining CLIPSeg for semantic understanding, SuperPoint for robust feature localization, and SAM for precise segmentation creates a powerful pipeline for few-shot tasks.
  • Graph Representation for Structure: Representing images as graphs, where nodes are interest points and edges capture spatial relationships, effectively encodes structural information and is permutation-invariant, aiding few-shot learning.
  • Efficient Few-Shot Learning: The approach achieves fast training times (under five minutes) and requires very few annotated samples, making it practical for adapting to new machinery or structures.
  • Synthetic-to-Real Generalization: The model demonstrates impressive generalization capabilities, performing well on real-world crane images after being trained solely on synthetic data. This highlights the value of domain randomization and robust foundation models.
  • Prompt Engineering is Key: The method's success hinges on an effective strategy for converting the GNN's output into SAM prompts, enabling SAM to accurately segment the identified parts.

Key Results

  • Synthetic Data Performance: Achieved J&F scores up to 90.1 for "Truck" granularity and 55.8 for "Low" granularity (8 classes) with just 25 support samples.
  • Synthetic-to-Real Generalization: Achieved a J&F score of 92.2 on real-world truck crane data using 10 synthetic support samples, demonstrating strong generalization.
  • DAVIS 2017 Performance: Achieved a J&F score of 71.5 in semi-supervised video segmentation with three support samples, outperforming several state-of-the-art methods not specifically designed for this task.
  • Fast Training: Training for a few-shot segmentation job with approximately three classes takes less than three minutes.

Limitations and Future Work

  • Interest Point Coverage: The reliance on interest points can sometimes lead to incomplete coverage of flat surfaces or low-contrast areas, potentially impacting segmentation accuracy.
  • Non-Deterministic SAM Input: The selection of SAM input points is currently non-deterministic, which could be improved with a learning-based approach.
  • Future Directions: The authors suggest exploring learning-based methods for SAM input selection and further investigating the impact of different foundation models and graph architectures. The potential for applying this framework to other complex structures beyond machinery is also implied.

Summary updated

BibTeX
@inproceedings{schwingshackl2025few,
  author = {Michael Schwingshackl and Fabio F. Oberweger and Markus Murschitz},
  title = {{Few-Shot Structure-Informed Machinery Part Segmentation with Foundation Models and Graph Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure