OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds

Ziyang Song, Bo Yang

Abstract

Identifying individual objects within 3D point clouds is a fundamental task for autonomous systems, yet it traditionally relies on extensive human-annotated datasets. In this paper, we present OGC (Object Geometry Consistency), the first unsupervised framework designed to segment multiple 3D objects from raw point clouds in a single forward pass. By leveraging the rigid motion patterns inherent in sequential data as a natural supervisory signal, OGC learns to discover object geometries without any human labels. Our approach integrates a multi-object segmentation network with a self-supervised scene flow estimator and a novel geometry consistency component. Through extensive evaluation across five diverse datasets, we demonstrate that OGC effectively identifies object instances in both indoor and complex outdoor scenarios, significantly narrowing the performance gap between unsupervised and supervised methods.

Introduction

3D object segmentation—the process of partitioning a point cloud into distinct object instances—is critical for robotics and autonomous driving. While supervised deep learning has achieved remarkable success in this domain, the "data hunger" of these models presents a significant bottleneck. Labeling 3D point clouds is an arduous, expensive process that does not scale easily to the infinite variety of real-world environments.

Existing unsupervised methods generally fall into two categories, both with significant limitations. Motion-based segmentation can identify moving objects but often fails at inference time if the objects are stationary (the "parked car" problem). Conversely, spatial clustering (like DBSCAN) relies on physical gaps between objects and fails when objects are in close proximity, such as a chair tucked under a table. OGC overcomes these hurdles by using motion as a teacher during training to learn geometric representations that persist even when objects are static during testing.

The OGC Framework

The core philosophy of OGC is that rigid objects move as a single unit. By observing how points move together over time, a model can infer the underlying structure of an object. The framework consists of three primary pillars:

1. Architecture: Transformer-Based Segmentation

The model utilizes a PointNet++ backbone to extract local geometric features from a single point cloud frame. These features are processed by a Transformer decoder that outputs multiple object masks in parallel. Unlike previous methods that require temporal sequences during inference, OGC’s architecture is designed to predict masks from a single static frame once trained.

2. Supervision: Self-Supervised Scene Flow

During the training phase, OGC employs an auxiliary scene flow estimator. This component calculates the displacement of every point between two consecutive frames without requiring ground-truth labels. This flow provides the "natural" supervision needed to understand which points belong to the same rigid body.

3. The Geometry Consistency Component

To translate motion signals into robust geometric understanding, OGC employs three specific loss functions:

  • Dynamic Rigid Consistency: This forces predicted masks to align with the estimated motion. If a group of points is identified as a single object, their collective motion must be mathematically explainable by a single rigid transformation (rotation and translation).
  • Object Shape Smoothness: This constraint assumes spatial continuity, suggesting that neighboring points in 3D space are likely to share the same object identity.
  • Object Shape Invariance: This is the key to generalization. It mandates that the model produce consistent object masks even if the input point cloud undergoes spatial transformations like rotation or scaling. This teaches the model that "object-ness" is a property of shape, not just position or motion.

Finally, an Iterative Optimization strategy is used to refine the results. The predicted masks are used to improve motion estimation via an object-aware Iterative Closest Point (ICP) algorithm, which in turn provides higher-quality supervision for the segmentation network.

Experimental Evaluation

OGC was evaluated across a spectrum of synthetic and real-world datasets, including SAPIEN (articulated objects), OGC-DR/DRSV (indoor scenes), and KITTI (outdoor traffic).

Performance Benchmarks

  • SAPIEN Dataset: OGC achieved a 55.6 AP (Average Precision), vastly outperforming classical clustering (DBSCAN: 6.3 AP) and existing motion-based methods (SSC: 9.5 AP).
  • KITTI-SF: In real-world outdoor scenarios, OGC reached 54.4 AP. This is particularly impressive as it approaches the performance of its fully supervised counterpart, OGC-sup, which scored 62.4 AP.
  • Flow Estimation: The iterative refinement process also benefited the auxiliary task. OGC reduced the End-point Error (EPE3D) on KITTI-SF to 6.72, surpassing the state-of-the-art unsupervised flow estimator, FlowStep3D (10.21).

Discussion and Limitations

While OGC represents a significant step forward, it faces specific challenges. Because it relies on motion for supervision, the model may struggle to identify object classes that never move during the training phase (e.g., buildings or trees), as there is no dynamic signal to define their boundaries. Additionally, the current framework assumes rigidity; while it handles articulated parts well, highly deformable objects like pedestrians remain a challenge. Finally, OGC is an instance segmenter; it identifies that an object exists without assigning a semantic label (e.g., "car" or "pedestrian").

Conclusion

OGC demonstrates that the physical laws of motion can serve as a powerful, free substitute for human labels in 3D vision. By bridging the gap between dynamic training and static inference, OGC provides a scalable path toward truly autonomous perception in complex, unlabeled environments.

Summary updated

Abstract

In this paper, we study the problem of 3D object segmentation from raw point clouds. Unlike all existing methods which usually require a large amount of human annotations for full supervision, we propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass, without needing any type of human annotations. The key to our approach is to fully leverage the dynamic motion patterns over sequential point clouds as supervision signals to automatically discover rigid objects. Our method consists of three major components, 1) the object segmentation network to directly estimate multi-object masks from a single point cloud frame, 2) the auxiliary self-supervised scene flow estimator, and 3) our core object geometry consistency component. By carefully designing a series of loss functions, we effectively take into account the multi-object rigid consistency and the object shape invariance in both temporal and spatial scales. This allows our method to truly discover the object geometry even in the absence of annotations. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation and general object segmentation in both indoor and the challenging outdoor scenarios.
BibTeX
@inproceedings{song2022ogc,
  author = {Ziyang Song and Bo Yang},
  title = {{OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=ecNbEOOtqBU},
}
Figure