3D Part Segmentation via Geometric Aggregation of 2D Visual Features

Marco Garosi, Riccardo Tedoldi, Davide Boscaini, Massimiliano Mancini, Nicu Sebe, Fabio Poiesi

Introduction

3D part segmentation—the task of identifying functional components of an object, such as the handle of a mug or the wing of an aircraft—is a cornerstone for applications in robotics and augmented reality. Traditional supervised models, while effective, are constrained by "closed-set" limitations, failing to generalize to unseen object categories or fine-grained parts not present in their training data.

Recent zero-shot approaches have leveraged Vision-Language Models (VLMs) like CLIP to bridge this gap. However, these methods typically suffer from two primary weaknesses: prompt sensitivity, where minor linguistic variations lead to catastrophic segmentation failures, and geometric blindness, where the 3D structure is ignored in favor of 2D visual matching, resulting in physically inconsistent part boundaries.

Methodology: The COPS Framework

To address these issues, we introduce COPS (COmprehensive model for Parts Segmentation), a training-free, zero-shot framework that prioritizes geometric consistency. The COPS pipeline decouples the physical decomposition of an object from the semantic labeling of its parts.

The COPS pipeline features a feature extractor (top) and the Geometric Feature Aggregation (GFA) module (bottom) to refine 3D point features.

The Five-Step Pipeline

  1. Multi-view Rendering: The 3D point cloud is rendered from multiple viewpoints (up to 48) to capture a comprehensive visual representation.
  2. Feature Extraction: Unlike previous methods that rely on CLIP's text-aligned visual features, COPS utilizes DINOv2, a self-supervised vision foundation model. DINOv2 provides dense, pixel-level features that capture structural nuances better than language-constrained models.
  3. Back-projection: The extracted 2D features are projected back onto the 3D points.
  4. Geometric Feature Aggregation (GFA): This core innovation refines noisy projections through two stages: spatially-consistent aggregation (smoothing local noise via physical proximity) and semantically-consistent aggregation (ensuring global consistency for similar features, such as the multiple legs of a chair).
  5. Decoupled Labeling: COPS first clusters refined features to identify physical parts. Only after the geometry is established does it use a VLM to assign semantic names to these clusters.

Key Insights and Results

The fundamental shift in COPS is the separation of part discovery from part naming. By using DINOv2, the model captures the underlying geometry of objects far more effectively than CLIP-based alternatives. The GFA module acts as a 3D "denoiser," enforcing physical reality on the visual features projected from 2D space.

Comparison of COPS against PointCLIPv2. COPS demonstrates superior mIoU and significantly higher robustness to varying prompt types.

Performance Benchmarks

COPS sets a new state-of-the-art for zero-shot 3D part segmentation across several benchmarks:

  • ShapeNetPart: Achieved 62.5% mIoU, outperforming the previous best (PointCLIPv2) by 10.7%.
  • Robustness: While PointCLIPv2's performance dropped to 5.1% mIoU when using only part names as prompts, COPS remained stable, proving its resilience to linguistic variation.
  • Real-World Application: On the ScanObjectNN dataset, which contains noisy real-world scans, COPS improved instance-average mIoU by 9.6% over existing baselines.

Conclusion

COPS demonstrates that 3D geometric reasoning is essential for robust part segmentation. By decoupling geometry from language and utilizing dense visual features from DINOv2, the framework overcomes the fragility of previous VLM-based methods. Future work will focus on eliminating the rendering bottleneck by applying geometric aggregation directly to raw sensor data like LiDAR or RGB-D streams.

Summary updated

BibTeX
@inproceedings{garosi2025d,
  author = {Marco Garosi and Riccardo Tedoldi and Davide Boscaini and Massimiliano Mancini and Nicu Sebe and Fabio Poiesi},
  title = {{3D Part Segmentation via Geometric Aggregation of 2D Visual Features}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00322},
}
Figure