Introduction
Vision-Language Models (vLLMs) have significantly advanced the ability of AI to interpret and segment visual data through natural language instructions. However, existing state-of-the-art models often struggle with complex, multi-target queries. LaSagnA (Language-based Segmentation Assistant) addresses these limitations by introducing a novel training recipe and sequence format that enables accurate, single-pass multi-object segmentation and rejection.
The Bottlenecks of Current vLLMs
Current models like LISA face two primary challenges when dealing with real-world instructions:
- Single-Target Limitation: Most models are optimized for one object per query. Requesting multiple items often requires multiple inference passes, which is computationally expensive and inefficient.
- The Hallucination Problem: Models frequently assume that every queried object exists in the image. This leads to "hallucinated" masks where the model forces a segmentation onto an unrelated object when the requested target is absent.
These issues typically stem from training data that lacks complexity, focusing primarily on single-target referring expressions rather than the multi-object scenarios common in human interaction.
Methodology: The LaSagnA Approach
LaSagnA builds upon the LLaVA architecture for vision-language understanding and the Segment Anything Model (SAM) for mask generation. The core innovation is a specialized training recipe and sequence format designed to handle complexity.
Single-Pass Sequence Format
Instead of generating a simple "It is <SEG>" response for each object, LaSagnA utilizes a structured list format: CategoryA <SEG>, CategoryB <NEG>, CategoryC <SEG>. The <SEG> token triggers the mask decoder for present objects, while the <NEG> token explicitly identifies absent categories. This "power of no" significantly reduces false positives and hallucinations.
Training Innovations
- Random Classes List: To prevent the model from relying on dataset-specific linguistic priors, it is trained with random subsets of categories, forcing it to rely on visual evidence.
- Sequence Augmentation: Negative samples (objects not in the image) are intentionally included in training queries to teach the model to reject non-existent targets.
- Order Consistency: The model is trained to respond in the exact order of the user's query, ensuring alignment between text and masks.
Results and Performance
LaSagnA demonstrates substantial improvements over previous models across various benchmarks:
- Semantic Segmentation: On the ADE20K benchmark, LaSagnA achieved a 42.0 mIoU, more than doubling the performance of LISA-7B (19.7).
- Reasoning Segmentation: The model outperformed LISA on complex reasoning tasks (47.2 vs 46.0), suggesting that mastering basic semantic segmentation enhances high-level reasoning capabilities.
- Efficiency: By processing $N$ targets in a single pass, LaSagnA offers a more scalable solution for complex real-world applications than models requiring $N$ separate inference cycles.
Conclusion
LaSagnA represents a significant step forward in making vLLMs more robust and efficient for complex visual tasks. By reframing semantic segmentation as a multi-target conversational query and incorporating explicit negative reasoning, it overcomes the primary bottlenecks of its predecessors. Future work will focus on improving performance in crowded scenes and extending the model to panoptic segmentation.
Summary updated