Abstract
We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. The task requires performing panoptic segmentation for not only known classes but also unknown ones that are not acknowledged during training. We investigate the challenges of the task and present a benchmark dataset on top of the existing COCO dataset. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies new classes with exemplars, which construct pseudo-ground-truths based on clustering, and augments the size of each class by adding new exemplars based on their similarity during training. We evaluate the proposed method on our benchmark and demonstrate the effectiveness of our proposals. The goal of our work is to draw the attention of the community to recognition in open-world scenarios.
Introduction: The Open-Set Challenge
Standard panoptic segmentation is a "closed-set" task: models are trained on a fixed list of categories (e.g., "car," "person," "grass") and are expected to ignore or misclassify anything else. However, real-world applications like autonomous driving encounter an infinite variety of objects. This paper addresses the Open-Set Panoptic Segmentation (OPS) task, where a model must segment both known classes and "unknown" objects it has never seen before.
The authors identify three major hurdles in moving to an open-set world:
- Definition of an Object: It is difficult to define where an "unknown" object begins and ends without prior category knowledge.
- Labeling Bias: In existing datasets, unknown objects are often lumped into a "background" or "void" category, teaching the model to ignore them.
- Evaluation: Measuring success is challenging when the ground truth for unknown objects is, by definition, missing from the training data.
The EOPSN Framework
The authors propose the Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN), building on the Panoptic FPN architecture. Inspired by psychological "exemplar theory"—the idea that we recognize new things by comparing them to specific memories of similar instances—EOPSN discovers new categories by analyzing unlabeled image regions.
The process involves two alternating stages:
- Clustering: The model analyzes "void" regions (areas not labeled as any known class). It uses $k$-means clustering on the features of object proposals in these regions to find groups of similar-looking "unknowns." These clusters become the initial "exemplars."
- Mining: As the model continues training, it compares new object proposals against these stored exemplars. If a proposal is highly similar to an existing exemplar, it is labeled as that specific unknown class. This allows the model to incrementally "grow" its knowledge of unknown categories over time.
To ensure the model doesn't misclassify these new discoveries as known classes, the authors modify the loss functions to provide "negative supervision."
Benchmark and Evaluation
To evaluate OPS, the authors created a new benchmark by reformatting the COCO (Common Objects in Context) dataset. They removed 5%, 10%, and 20% of the "thing" classes (like "car" or "keyboard") from the training set, treating them as the "unknown" classes during testing. This allows them to use the original COCO labels as a hidden ground truth to measure how well the model discovered classes it wasn't supposed to know.
Key Results
EOPSN significantly outperformed baselines that relied purely on "objectness" scores (the general probability that a region contains something).
- Quantitative: On the COCO 20% split, EOPSN achieved an Unknown Panoptic Quality (PQ) of 11.3, compared to the baseline's 7.8.
- Qualitative: The model successfully discovered and segmented objects like "stop signs" and "sinks" even when it was never told those categories existed. Interestingly, it even discovered sub-categories not in the original COCO labels, such as individual keys on a keyboard.
Conclusion and Future Work
While EOPSN demonstrates that "void" regions are a goldmine for discovering the unknown, limitations remain. The clustering stage is computationally expensive, and the current method focuses primarily on "thing" (instance) segmentation rather than "stuff" (amorphous regions like sky). Future work will explore more efficient discovery mechanisms and broader semantic understanding of the open world.
Summary updated