Introduction
3D part segmentation—the task of identifying functional components of an object, such as the handle of a mug or the wing of an aircraft—is a cornerstone for applications in robotics and augmented reality. Traditional supervised models, while effective, are constrained by "closed-set" limitations, failing to generalize to unseen object categories or fine-grained parts not present in their training data.
Recent zero-shot approaches have leveraged Vision-Language Models (VLMs) like CLIP to bridge this gap. However, these methods typically suffer from two primary weaknesses: prompt sensitivity, where minor linguistic variations lead to catastrophic segmentation failures, and geometric blindness, where the 3D structure is ignored in favor of 2D visual matching, resulting in physically inconsistent part boundaries.
Methodology: The COPS Framework
To address these issues, we introduce COPS (COmprehensive model for Parts Segmentation), a training-free, zero-shot framework that prioritizes geometric consistency. The COPS pipeline decouples the physical decomposition of an object from the semantic labeling of its parts.
The Five-Step Pipeline
- Multi-view Rendering: The 3D point cloud is rendered from multiple viewpoints (up to 48) to capture a comprehensive visual representation.
- Feature Extraction: Unlike previous methods that rely on CLIP's text-aligned visual features, COPS utilizes DINOv2, a self-supervised vision foundation model. DINOv2 provides dense, pixel-level features that capture structural nuances better than language-constrained models.
- Back-projection: The extracted 2D features are projected back onto the 3D points.
- Geometric Feature Aggregation (GFA): This core innovation refines noisy projections through two stages: spatially-consistent aggregation (smoothing local noise via physical proximity) and semantically-consistent aggregation (ensuring global consistency for similar features, such as the multiple legs of a chair).
- Decoupled Labeling: COPS first clusters refined features to identify physical parts. Only after the geometry is established does it use a VLM to assign semantic names to these clusters.
Key Insights and Results
The fundamental shift in COPS is the separation of part discovery from part naming. By using DINOv2, the model captures the underlying geometry of objects far more effectively than CLIP-based alternatives. The GFA module acts as a 3D "denoiser," enforcing physical reality on the visual features projected from 2D space.
Performance Benchmarks
COPS sets a new state-of-the-art for zero-shot 3D part segmentation across several benchmarks:
- ShapeNetPart: Achieved 62.5% mIoU, outperforming the previous best (PointCLIPv2) by 10.7%.
- Robustness: While PointCLIPv2's performance dropped to 5.1% mIoU when using only part names as prompts, COPS remained stable, proving its resilience to linguistic variation.
- Real-World Application: On the ScanObjectNN dataset, which contains noisy real-world scans, COPS improved instance-average mIoU by 9.6% over existing baselines.
Conclusion
COPS demonstrates that 3D geometric reasoning is essential for robust part segmentation. By decoupling geometry from language and utilizing dense visual features from DINOv2, the framework overcomes the fragility of previous VLM-based methods. Future work will focus on eliminating the rendering bottleneck by applying geometric aggregation directly to raw sensor data like LiDAR or RGB-D streams.
Summary updated