Exemplar-Based Open-Set Panoptic Segmentation Network

Jaedong Hwang, Seoung Wug Oh, Joon-Young Lee, Bohyung Han

Abstract

We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. The task requires performing panoptic segmentation for not only known classes but also unknown ones that are not acknowledged during training. We investigate the challenges of the task and present a benchmark dataset on top of the existing COCO dataset. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies new classes with exemplars, which construct pseudo-ground-truths based on clustering, and augments the size of each class by adding new exemplars based on their similarity during training. We evaluate the proposed method on our benchmark and demonstrate the effectiveness of our proposals. The goal of our work is to draw the attention of the community to recognition in open-world scenarios.

Introduction: The Open-Set Challenge

Standard panoptic segmentation is a "closed-set" task: models are trained on a fixed list of categories (e.g., "car," "person," "grass") and are expected to ignore or misclassify anything else. However, real-world applications like autonomous driving encounter an infinite variety of objects. This paper addresses the Open-Set Panoptic Segmentation (OPS) task, where a model must segment both known classes and "unknown" objects it has never seen before.

The authors identify three major hurdles in moving to an open-set world:

  1. Definition of an Object: It is difficult to define where an "unknown" object begins and ends without prior category knowledge.
  2. Labeling Bias: In existing datasets, unknown objects are often lumped into a "background" or "void" category, teaching the model to ignore them.
  3. Evaluation: Measuring success is challenging when the ground truth for unknown objects is, by definition, missing from the training data.

The EOPSN Framework

The authors propose the Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN), building on the Panoptic FPN architecture. Inspired by psychological "exemplar theory"—the idea that we recognize new things by comparing them to specific memories of similar instances—EOPSN discovers new categories by analyzing unlabeled image regions.

EOPSN Architecture: The model alternates between clustering features from void regions to create exemplars and mining new images to identify and refine these unknown classes.

The process involves two alternating stages:

  • Clustering: The model analyzes "void" regions (areas not labeled as any known class). It uses $k$-means clustering on the features of object proposals in these regions to find groups of similar-looking "unknowns." These clusters become the initial "exemplars."
  • Mining: As the model continues training, it compares new object proposals against these stored exemplars. If a proposal is highly similar to an existing exemplar, it is labeled as that specific unknown class. This allows the model to incrementally "grow" its knowledge of unknown categories over time.

To ensure the model doesn't misclassify these new discoveries as known classes, the authors modify the loss functions to provide "negative supervision."

Benchmark and Evaluation

To evaluate OPS, the authors created a new benchmark by reformatting the COCO (Common Objects in Context) dataset. They removed 5%, 10%, and 20% of the "thing" classes (like "car" or "keyboard") from the training set, treating them as the "unknown" classes during testing. This allows them to use the original COCO labels as a hidden ground truth to measure how well the model discovered classes it wasn't supposed to know.

Open-Set Evaluation Logic: The task requires identifying known classes while correctly grouping and labeling unknown objects without explicit ground truth during training.

Key Results

EOPSN significantly outperformed baselines that relied purely on "objectness" scores (the general probability that a region contains something).

  • Quantitative: On the COCO 20% split, EOPSN achieved an Unknown Panoptic Quality (PQ) of 11.3, compared to the baseline's 7.8.
  • Qualitative: The model successfully discovered and segmented objects like "stop signs" and "sinks" even when it was never told those categories existed. Interestingly, it even discovered sub-categories not in the original COCO labels, such as individual keys on a keyboard.

Conclusion and Future Work

While EOPSN demonstrates that "void" regions are a goldmine for discovering the unknown, limitations remain. The clustering stage is computationally expensive, and the current method focuses primarily on "thing" (instance) segmentation rather than "stuff" (amorphous regions like sky). Future work will explore more efficient discovery mechanisms and broader semantic understanding of the open world.

Summary updated

Abstract

We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. The task requires to perform panoptic segmentation for not only known classes but also unknown ones that are not acknowledged during training. We investigate challenges of the task and present a benchmark dataset on top of an existing dataset, COCO. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies a new class with exemplars, which constructs pseudo-ground-truths, based on clustering and augments the size of each class by adding new exemplars based on their similarity during training. We evaluate the proposed method on our benchmark and demonstrate the effectiveness of our proposals. The goal of our work is to draw the attention of the community to the recognition in open-world scenarios.
BibTeX
@inproceedings{hwang2021exemplar,
  author = {Jaedong Hwang and Seoung Wug Oh and Joon-Young Lee and Bohyung Han},
  title = {{Exemplar-Based Open-Set Panoptic Segmentation Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=Rox-ym8wc1},
}
Figure