Abstract
In the past few years, we have seen great progress in perception algorithms, particular through the use of deep learning. However, most existing approaches focus on a few categories of interest, which represent only a small fraction of the potential categories that robots need to handle in the real-world. Thus, identifying objects from unknown classes remains a challenging yet crucial task. In this paper, we develop a novel open-set instance segmentation algorithm for point clouds which can segment objects from both known and unknown classes in a holistic way. Our method uses a deep convolutional network to project points into a category-agnostic embedding space in which they can be clustered into instances irrespective of their semantics. Experiments on two large-scale self-driving datasets validate the effectiveness of our proposed method.
Introduction
Most current autonomous driving systems operate under a "closed-set" assumption, meaning they are trained to recognize a specific list of categories such as cars, pedestrians, and cyclists. However, real-world environments are unpredictable. When a sensor encounters an object outside this predefined list—such as a rare animal or fallen debris—traditional systems may ignore it, misclassify it, or fail to segment it as a distinct entity.
To operate safely, robots must possess the capacity for perceptual grouping: the ability to recognize that a cluster of visual elements belongs to a single "thing" without necessarily knowing its semantic label. The challenge lies in building a deep learning model that generalizes the concept of "objectness" to unknown classes without over-relying on semantic labels that it will not have during inference.
Methodology
The proposed Open-Set Instance Segmentation (OSIS) network addresses this by projecting LiDAR point clouds into a category-agnostic embedding space. The architecture consists of three primary components:
- Shared Backbone: A 2D convolutional feature pyramid network processing LiDAR point clouds in a Bird’s Eye View (BEV).
- Detection Head: Identifies "anchors" for known "thing" classes to provide top-down guidance for the segmentation process.
- Embedding Head: Projects every point into an embedding space where points from the same instance are trained to be close together, regardless of whether the object is known or unknown.
The inference pipeline follows a two-stage approach. First, it uses "prototypes" for known classes to filter out and segment recognized objects. Second, for the remaining "uncertain" points, the system performs DBSCAN clustering in a combined space of 3D coordinates and learned embeddings to identify and group "unknown" instances.
Experimental Evaluation
The model was evaluated on two large-scale datasets: TOR4D, containing 6,500 scenarios with manually annotated unknown objects, and Rare4D, a curated dataset featuring 289 unique rare objects like horses, tractors, and forklifts that are almost never seen in daily driving.
Key Results
OSIS significantly outperformed existing baselines in identifying unknown instances. On the Rare4D dataset, OSIS achieved an Unknown Quality (UQ) score of 62.5, surpassing the best baseline (57.9). Interestingly, the open-set approach also improved performance on known classes; on TOR4D, OSIS achieved a Recognition Quality (RQ) of 84.3 for known things, compared to 81.4 for the Panoptic3D baseline.
Qualitative analysis demonstrates that OSIS can correctly segment novel objects, such as a horse on a road, as a single "unknown" instance, whereas baseline models often produce fragmented or noisy results. By forcing the network to learn embeddings based on "instance-ness" rather than "class-ness," the model learns a more fundamental geometric representation of what constitutes an object.
Conclusion
OSIS represents a significant step toward robust open-set perception for autonomous vehicles by marrying top-down detection with bottom-up learned embeddings. While the model occasionally misclassifies complex unknown objects, its ability to generalize "objectness" is a major advancement over closed-set systems. Future work will explore integrating temporal consistency and motion cues—such as the fact that clusters of points moving together likely belong to one object—to further refine perceptual grouping.
Summary updated