TL;DR
This paper introduces ROSE, a novel Large Multimodal Model (LMM) for open-set dense segmentation that can generate categories and masks without predefined prompts. It achieves this by processing images patch-by-patch and leveraging a generative instruction-response paradigm, significantly advancing the flexibility and autonomy of segmentation models.
What Problem Does This Paper Address?
Traditional image segmentation models are limited by their reliance on fixed, predefined categories learned during training. While recent advancements in models like CLIP and Large Multimodal Models (LMMs) have enabled "open-vocabulary" segmentation, they still require users to provide category candidates. Furthermore, many existing LMMs for segmentation produce sparse predictions, limiting their utility in applications demanding fine-grained, pixel-level understanding. This paper tackles the challenge of creating a segmentation model that can not only predict dense masks but also autonomously generate novel categories for objects it encounters, moving beyond the constraints of closed-set recognition.
Methods
ROSE introduces several key innovations:
- Patch-wise Perception: Instead of processing the entire image at once, ROSE divides the input image into smaller patches. Each patch is treated as an independent "region of interest candidate." This approach helps overcome the long-range spatial dependency issues that can arise when LMMs process long sequences of segmentation tokens, leading to more stable and detailed predictions. For each patch, the model predicts an "objectness score" (likelihood of containing an object), a "mask embedding" (for generating the mask), and a "category embedding" (for identifying the object's class).
- Generative Open-Category Generation: ROSE moves away from simply selecting from predefined categories. It employs an "instruction-response paradigm" where the category embedding is used as a linguistic feature. The model then generates category names in a free-form, language-driven manner, allowing it to identify and name previously unseen objects.
- Conversation-based Refinement: To further enhance accuracy and detail, ROSE incorporates a "conversation-based refinement" mechanism. This allows the model to iteratively refine its segmentation masks and category predictions based on user-provided textual feedback, mimicking a human-like correction process.
Datasets
The paper evaluates ROSE on several standard segmentation benchmarks:
- Semantic Segmentation: ADE-20k, COCO-Stuff, and Mapillary.
- Instance Segmentation: COCO.
- Referring Segmentation: RefCLEF, RefCOCO, RefCOCO+, and RefCOCOg.
The paper does not introduce new datasets but focuses on adapting existing ones for its open-set, dense segmentation task.
Key Insights and Ideas
- Patching for Stability: Treating image patches as independent units is a clever way to manage the complexity of dense prediction in LMMs, preventing information dilution that can occur with long token sequences.
- Generative Categories are Key: Moving from category selection to category generation is a crucial step towards true open-set understanding, allowing models to adapt to novel visual concepts without explicit training data for them.
- Conversational Refinement is Powerful: The ability to iteratively refine predictions based on feedback is a significant step towards more robust and user-friendly segmentation systems, mimicking human correction processes.
Key Results
ROSE demonstrates competitive performance across various segmentation tasks.
- Semantic Segmentation: ROSE achieves a significant performance boost with its Conversation-based Segmentation Refinement (CSR) mechanism, reaching 51.6 mIoU on ADE-20k, outperforming previous SOTA models.
- Instance Segmentation: ROSE achieves a comparable performance of 34.4 mAP on COCO, leveraging its objectness score for confidence.
- Referring Segmentation: ROSE achieves multiple SOTA results on RefCOCO/+/g datasets, showcasing its strong referring ability, even when compared to larger models.
- Refinement Gains: The CSR mechanism provides substantial improvements, including an 8.6 mIoU boost in semantic segmentation and 2.0 mAP improvements in instance segmentation.
Qualitative results show ROSE successfully generating categories and dense masks for objects in both in-domain (COCO) and cross-domain (crayon drawings, clip art) scenarios.
Limitations and Future Work
- Granularity Differences: The paper acknowledges that inherent limitations in evaluation methods can lead to misclassifications due to granularity differences (e.g., "plant" vs. "tree"). This is identified as an important area for future research.
- Benchmark Limitations: The authors note that while their approach advances open-set dense segmentation, the lack of a comprehensive benchmark specifically for diverse open-set scenarios limits their ability to fully evaluate the model's performance.
- Future Work: The paper suggests that their work opens a new direction for future open-set segmentation in diverse and dynamic environments. The refinement mechanism, in particular, shows promise for improving accuracy and handling complex visual scenes.
Summary updated