Abstract
We extend panoptic segmentation to the open-world and introduce an open-set panoptic segmentation (OPS) task. This task requires performing panoptic segmentation for not only known classes but also unknown ones that have not been acknowledged during training. We investigate the practical challenges of the task and construct a benchmark on top of an existing dataset, COCO. In addition, we propose a novel exemplar-based open-set panoptic segmentation network (EOPSN) inspired by exemplar theory. Our approach identifies a new class based on exemplars, which are identified by clustering and employed as pseudo-ground-truths. The size of each class increases by mining new exemplars based on the similarities to the existing ones associated with the class. We evaluate EOPSN on the proposed benchmark and demonstrate the effectiveness of our proposals.
Introduction: From Closed to Open Worlds
Standard panoptic segmentation—the task of assigning a semantic label (e.g., "car" or "sky") and an instance ID (e.g., "car #1" vs "car #2") to every pixel—operates under a "closed-world" assumption. Models are typically trained on a fixed set of classes and either fail or ignore anything outside that list. However, in real-world applications like autonomous driving or robotics, systems inevitably encounter objects they were never trained to recognize.
Current models usually lump these "unknown unknowns" into a generic "void" or "background" category. The Open-Set Panoptic Segmentation (OPS) task addresses this by requiring the model to not only detect unknown objects but also group similar-looking unknown objects into distinct pseudo-classes rather than treating them as background noise.
Methodology: Exemplar-Based Discovery
The proposed Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN) builds on the Panoptic FPN architecture, adding a dynamic discovery mechanism inspired by psychological "exemplar theory." This theory suggests that humans categorize new entities by comparing them to specific memories of similar instances.
The EOPSN approach operates in two main stages:
- Clustering (Discovery): The model examines "void" regions—areas not labeled in the training data—and extracts features from potential object proposals. Using k-means clustering, it groups these features. Tightly packed groups with high "objectness" scores are designated as new "unknown classes," and these instances are stored as exemplars.
- Mining (Expansion): As the model processes more images, it compares new object proposals to stored exemplars. If a new object exhibits high similarity to an existing exemplar, it is "mined" and assigned to that specific unknown class.
To ensure feasibility, the authors assume that unknown classes are "things" (distinct objects) rather than "stuff" (amorphous regions), and that unknowns primarily appear in regions labeled as "void" during training.
Experiments and Evaluation
To evaluate the model, the researchers established the COCO OPS Benchmark by reformatting the COCO dataset. They simulated an open-set environment by "hiding" specific percentages (5%, 10%, or 20%) of "thing" classes during training to be treated as unknowns during testing.
Evaluation is performed using Panoptic Quality (PQ), which integrates segmentation accuracy (SQ) and recognition accuracy (RQ). In open-set scenarios, the model must correctly distinguish between known classes, unknown classes, and unlabeled regions.
Key Results
- Superior Unknown Recognition: On the 20% unknown split, EOPSN achieved a PQ of 11.3 for unknown classes, significantly outperforming the strongest baseline of 7.8 PQ.
- Stability of Known Classes: The model maintained competitive performance on known classes (approx. 37.4 PQ), demonstrating that learning new categories does not necessitate catastrophic forgetting of old ones.
- Recognition Quality (RQ) Gains: EOPSN nearly doubled the baseline's performance in RQ, highlighting its superior ability to group instances of the same unknown class correctly.
Conclusion
EOPSN demonstrates the potential for AI to move beyond fixed vocabularies. By treating the "void" as a source of information rather than noise, the model can identify objects outside the original COCO vocabulary, such as "keys" on a keyboard. Future work will focus on extending this capability to "stuff" classes—amorphous regions like terrain—and better integrating semantic segmentation branches to improve the discovery of diverse unknown categories.
Summary updated