Generalized Class Discovery in Instance Segmentation

Cuong Manh Hoang, Yeejin Lee, Byeongkeun KangAAAI 2025 · DOI

TL;DR

This paper tackles the challenging problem of instance segmentation in "open-world" scenarios where new object categories can appear in unlabeled data. They propose a novel framework that combines instance-wise temperature assignment for contrastive learning, class-wise dynamic reliability criteria for pseudo-labeling, and a soft attention module to effectively segment both known and newly discovered objects, outperforming prior state-of-the-art methods.

What Problem Does This Paper Address?

Traditional instance segmentation models operate under a "closed-world" assumption, meaning they can only identify and segment objects belonging to the predefined classes seen during training. This is a significant limitation in real-world applications where new object categories are constantly emerging. Generalized Class Discovery (GCD) aims to address this by enabling models to discover novel classes from unlabeled data and segment instances of both known and novel categories. However, existing GCD methods often struggle with the inherent long-tailed distribution of objects in real-world datasets (e.g., many cars, few ashtrays), leading to imbalanced class representations and poor performance on tail classes. This paper specifically targets GCD for instance segmentation, a more complex task than image classification, and aims to overcome the challenges posed by imbalanced data distributions.

Methods

The proposed method introduces several key innovations:

  • Instance-wise Temperature Assignment (ITA): To combat the imbalanced class distributions in contrastive learning, ITA dynamically adjusts the temperature parameter for each instance. This allows for stronger discrimination among "head" classes (frequently occurring) while encouraging more flexible discrimination for "tail" classes (infrequently occurring). This is achieved by estimating an instance's "headness" score based on its neighborhood in the embedding space.
  • Class-wise Reliability Criteria and Dynamic Learning (RDL): Standard pseudo-labeling methods often use fixed reliability criteria, which can discard too many samples from tail classes. RDL introduces class-specific reliability thresholds and dynamically adjusts them throughout training. This allows the model to leverage diverse samples in early training stages and focus on more reliable pseudo-labels in later stages, improving robustness for tail classes.
  • Soft Attention Module (SAM): To better capture object-specific features and handle noisy pseudo-masks generated for unlabeled data, a soft attention module is incorporated. This module uses pooled feature maps and depth reduction to efficiently encode spatial relationships and generate an attention map that refines feature representations.
  • Deep Clustering: The method also incorporates a deep clustering approach for unsupervised learning on unlabeled data, avoiding the need for experimentally selected prior distributions and handling imbalanced data.

Datasets

The experiments were conducted on two benchmark settings:

  • COCOhalf + LVIS: This setting uses 80 COCO classes as known classes and aims to discover 1,123 disjoint LVIS classes.
  • LVIS + VG: Here, 1,203 LVIS classes are treated as known classes, and the goal is to discover 2,726 disjoint classes from the Visual Genome (VG) dataset.

The paper utilizes subsets of the COCO, LVIS, and Visual Genome datasets for training and evaluation.

Key Insights and Ideas

  • Tail classes are the Achilles' heel: The paper highlights that imbalanced data distributions are a major hurdle for GCD, particularly for instance segmentation. Simply applying standard methods designed for balanced datasets leads to poor performance on rare object categories.
  • Dynamic adaptation is crucial: Both temperature assignment in contrastive learning and the reliability of pseudo-labels benefit significantly from dynamic, instance- or class-wise adaptation rather than static, global settings.
  • Attention can refine noisy signals: The soft attention module proves effective in leveraging potentially noisy pseudo-masks from unlabeled data, demonstrating its utility in real-world scenarios where perfect annotations are unavailable.
  • GCD for instance segmentation is a complex but solvable problem: The proposed method shows that by carefully addressing data imbalance and pseudo-label quality, significant progress can be made in discovering and segmenting novel objects.

Key Results

The proposed method consistently outperforms previous state-of-the-art methods across various metrics (mAPall, mAPknown, mAPnovel) on both the COCOhalf + LVIS and LVIS + VG settings. For instance, on the COCOhalf + LVIS setting, the proposed method achieves an mAPall of 12.85, significantly higher than the next best method (NCDLR+) at 9.42. Qualitative results also show superior performance in segmenting novel objects. Ablation studies confirm that each proposed component (ITA, RDL, SAM) contributes to the overall performance improvement.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Sequential data: The current framework assumes the availability of all labeled and unlabeled data at the beginning of training. It is suboptimal for scenarios where data is provided sequentially, such as in robot navigation.
  • Prior knowledge of class count: The method, like most prior works, assumes prior knowledge of the total number of classes (known + novel).

Future work could explore methods that can handle sequential data and automatically determine the number of novel classes.

Summary updated

BibTeX
@inproceedings{hoang2025generalized,
  author = {Cuong Manh Hoang and Yeejin Lee and Byeongkeun Kang},
  title = {{Generalized Class Discovery in Instance Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32362},
}
Figure