LaSagnA: Language-based Segmentation Assistant for Complex Queries

Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, Lin MaarXiv 2024 · DOI · arXiv

Introduction

Vision-Language Models (vLLMs) have significantly advanced the ability of AI to interpret and segment visual data through natural language instructions. However, existing state-of-the-art models often struggle with complex, multi-target queries. LaSagnA (Language-based Segmentation Assistant) addresses these limitations by introducing a novel training recipe and sequence format that enables accurate, single-pass multi-object segmentation and rejection.

The Bottlenecks of Current vLLMs

Current models like LISA face two primary challenges when dealing with real-world instructions:

  1. Single-Target Limitation: Most models are optimized for one object per query. Requesting multiple items often requires multiple inference passes, which is computationally expensive and inefficient.
  2. The Hallucination Problem: Models frequently assume that every queried object exists in the image. This leads to "hallucinated" masks where the model forces a segmentation onto an unrelated object when the requested target is absent.

These issues typically stem from training data that lacks complexity, focusing primarily on single-target referring expressions rather than the multi-object scenarios common in human interaction.

Methodology: The LaSagnA Approach

LaSagnA builds upon the LLaVA architecture for vision-language understanding and the Segment Anything Model (SAM) for mask generation. The core innovation is a specialized training recipe and sequence format designed to handle complexity.

Single-Pass Sequence Format

Instead of generating a simple "It is <SEG>" response for each object, LaSagnA utilizes a structured list format: CategoryA <SEG>, CategoryB <NEG>, CategoryC <SEG>. The <SEG> token triggers the mask decoder for present objects, while the <NEG> token explicitly identifies absent categories. This "power of no" significantly reduces false positives and hallucinations.

Training Innovations

  • Random Classes List: To prevent the model from relying on dataset-specific linguistic priors, it is trained with random subsets of categories, forcing it to rely on visual evidence.
  • Sequence Augmentation: Negative samples (objects not in the image) are intentionally included in training queries to teach the model to reject non-existent targets.
  • Order Consistency: The model is trained to respond in the exact order of the user's query, ensuring alignment between text and masks.

Results and Performance

LaSagnA demonstrates substantial improvements over previous models across various benchmarks:

  • Semantic Segmentation: On the ADE20K benchmark, LaSagnA achieved a 42.0 mIoU, more than doubling the performance of LISA-7B (19.7).
  • Reasoning Segmentation: The model outperformed LISA on complex reasoning tasks (47.2 vs 46.0), suggesting that mastering basic semantic segmentation enhances high-level reasoning capabilities.
  • Efficiency: By processing $N$ targets in a single pass, LaSagnA offers a more scalable solution for complex real-world applications than models requiring $N$ separate inference cycles.

Conclusion

LaSagnA represents a significant step forward in making vLLMs more robust and efficient for complex visual tasks. By reframing semantic segmentation as a multi-target conversational query and incorporating explicit negative reasoning, it overcomes the primary bottlenecks of its predecessors. Future work will focus on improving performance in crowded scenes and extending the model to panoptic segmentation.

Summary updated

Abstract

Recent advancements have empowered Large Language Models for Vision (vLLMs) to generate detailed perceptual outcomes, including bounding boxes and masks. Nonetheless, there are two constraints that restrict the further application of these vLLMs: the incapability of handling multiple targets per query and the failure to identify the absence of query objects in the image. In this study, we acknowledge that the main cause of these problems is the insufficient complexity of training queries. Consequently, we define the general sequence format for complex queries. Then we incorporate a semantic segmentation task in the current pipeline to fulfill the requirements of training data. Furthermore, we present three novel strategies to effectively handle the challenges arising from the direct integration of the proposed format. The effectiveness of our model in processing complex queries is validated by the comparable results with conventional methods on both close-set and open-set semantic segmentation datasets. Additionally, we outperform a series of vLLMs in reasoning and referring segmentation, showcasing our model's remarkable capabilities. We release the code at https://github.com/congvvc/LaSagnA.
BibTeX
@misc{wei2024lasagna,
  author = {Cong Wei and Haoxian Tan and Yujie Zhong and Yujiu Yang and Lin Ma},
  title = {{LaSagnA: Language-based Segmentation Assistant for Complex Queries}},
  year = {2024},
  eprint = {2404.08506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.08506},
}
Figure