ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model

Kunyang Han, Yibo Hu, Mengxue Qu, Hailin Shi, Yao Zhao, Yunchao WeiarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces ROSE, a novel Large Multimodal Model (LMM) for open-set dense segmentation that can generate categories and masks without predefined prompts. It achieves this by processing images patch-by-patch and leveraging a generative instruction-response paradigm, significantly advancing the flexibility and autonomy of segmentation models.

What Problem Does This Paper Address?

Traditional image segmentation models are limited by their reliance on fixed, predefined categories learned during training. While recent advancements in models like CLIP and Large Multimodal Models (LMMs) have enabled "open-vocabulary" segmentation, they still require users to provide category candidates. Furthermore, many existing LMMs for segmentation produce sparse predictions, limiting their utility in applications demanding fine-grained, pixel-level understanding. This paper tackles the challenge of creating a segmentation model that can not only predict dense masks but also autonomously generate novel categories for objects it encounters, moving beyond the constraints of closed-set recognition.

Methods

ROSE introduces several key innovations:

  • Patch-wise Perception: Instead of processing the entire image at once, ROSE divides the input image into smaller patches. Each patch is treated as an independent "region of interest candidate." This approach helps overcome the long-range spatial dependency issues that can arise when LMMs process long sequences of segmentation tokens, leading to more stable and detailed predictions. For each patch, the model predicts an "objectness score" (likelihood of containing an object), a "mask embedding" (for generating the mask), and a "category embedding" (for identifying the object's class).
  • Generative Open-Category Generation: ROSE moves away from simply selecting from predefined categories. It employs an "instruction-response paradigm" where the category embedding is used as a linguistic feature. The model then generates category names in a free-form, language-driven manner, allowing it to identify and name previously unseen objects.
  • Conversation-based Refinement: To further enhance accuracy and detail, ROSE incorporates a "conversation-based refinement" mechanism. This allows the model to iteratively refine its segmentation masks and category predictions based on user-provided textual feedback, mimicking a human-like correction process.

Datasets

The paper evaluates ROSE on several standard segmentation benchmarks:

  • Semantic Segmentation: ADE-20k, COCO-Stuff, and Mapillary.
  • Instance Segmentation: COCO.
  • Referring Segmentation: RefCLEF, RefCOCO, RefCOCO+, and RefCOCOg.

The paper does not introduce new datasets but focuses on adapting existing ones for its open-set, dense segmentation task.

Key Insights and Ideas

  • Patching for Stability: Treating image patches as independent units is a clever way to manage the complexity of dense prediction in LMMs, preventing information dilution that can occur with long token sequences.
  • Generative Categories are Key: Moving from category selection to category generation is a crucial step towards true open-set understanding, allowing models to adapt to novel visual concepts without explicit training data for them.
  • Conversational Refinement is Powerful: The ability to iteratively refine predictions based on feedback is a significant step towards more robust and user-friendly segmentation systems, mimicking human correction processes.

Key Results

ROSE demonstrates competitive performance across various segmentation tasks.

  • Semantic Segmentation: ROSE achieves a significant performance boost with its Conversation-based Segmentation Refinement (CSR) mechanism, reaching 51.6 mIoU on ADE-20k, outperforming previous SOTA models.
  • Instance Segmentation: ROSE achieves a comparable performance of 34.4 mAP on COCO, leveraging its objectness score for confidence.
  • Referring Segmentation: ROSE achieves multiple SOTA results on RefCOCO/+/g datasets, showcasing its strong referring ability, even when compared to larger models.
  • Refinement Gains: The CSR mechanism provides substantial improvements, including an 8.6 mIoU boost in semantic segmentation and 2.0 mAP improvements in instance segmentation.

Qualitative results show ROSE successfully generating categories and dense masks for objects in both in-domain (COCO) and cross-domain (crayon drawings, clip art) scenarios.

Limitations and Future Work

  • Granularity Differences: The paper acknowledges that inherent limitations in evaluation methods can lead to misclassifications due to granularity differences (e.g., "plant" vs. "tree"). This is identified as an important area for future research.
  • Benchmark Limitations: The authors note that while their approach advances open-set dense segmentation, the lack of a comprehensive benchmark specifically for diverse open-set scenarios limits their ability to fully evaluate the model's performance.
  • Future Work: The paper suggests that their work opens a new direction for future open-set segmentation in diverse and dynamic environments. The refinement mechanism, in particular, shows promise for improving accuracy and handling complex visual scenes.

Summary updated

Abstract

Advances in CLIP and large multimodal models (LMMs) have enabled open-vocabulary and free-text segmentation, yet existing models still require predefined category prompts, limiting free-form category self-generation. Most segmentation LMMs also remain confined to sparse predictions, restricting their applicability in open-set environments. In contrast, we propose ROSE, a Revolutionary Open-set dense SEgmentation LMM, which enables dense mask prediction and open-category generation through patch-wise perception. Our method treats each image patch as an independent region of interest candidate, enabling the model to predict both dense and sparse masks simultaneously. Additionally, a newly designed instruction-response paradigm takes full advantage of the generation and generalization capabilities of LMMs, achieving category prediction independent of closed-set constraints or predefined categories. To further enhance mask detail and category precision, we introduce a conversation-based refinement paradigm, integrating the prediction result from previous step with textual prompt for revision. Extensive experiments demonstrate that ROSE achieves competitive performance across various segmentation tasks in a unified framework. Code will be released.
BibTeX
@misc{han2024rose,
  author = {Kunyang Han and Yibo Hu and Mengxue Qu and Hailin Shi and Yao Zhao and Yunchao Wei},
  title = {{ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model}},
  year = {2024},
  eprint = {2412.00153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.00153},
}
Figure