Reasoning Segmentation with Multimodal Large Language Models

This direction is split into other directions: token-triggered segmentation. Its last version stays readable and citable here.

Version 1 of 1149 papers179 conceptsfrozen 2026-10-04

Traditional image segmentation is a "naming" task: a user provides a label like "dog," and the model finds the pixels. However, real-world interaction often requires "reasoning" tasks. If a user asks to "segment the food with the most Vitamin C," the model cannot rely on a simple label lookup. It must first use world knowledge to identify the orange in the bowl and then locate its boundaries.

This shift from explicit naming to implicit reasoning is the core of Reasoning Segmentation (RS), a direction pioneered by Xin Lai and colleagues in LISA: Reasoning Segmentation via Large Language Model. By integrating the logical depth of Large Language Models (LLMs) with the spatial precision of segmentation decoders, researchers are enabling AI to "point" to objects identified through complex multi-step thought.

The Token as a Bridge

The fundamental challenge in reasoning segmentation is architectural: how do you translate a high-level linguistic conclusion into a low-level pixel mask? The dominant solution across recent literature is the introduction of a specialized segmentation token (often denoted as <SEG>).

In models like LISA and GLaMM, the LLM treats "segmenting" as just another word in its vocabulary. When the model determines that a specific object needs to be grounded, it generates the <SEG> token. The intuition here is that the hidden state embedding of this token—the internal mathematical representation the LLM holds just before outputting the word—contains the "distilled essence" of the object the model is thinking about.

This embedding is then projected into the feature space of a dedicated mask decoder, such as the Segment Anything Model (SAM). As Zhuofan Xia demonstrates in GSVA, this allows the LLM's internal attention to route specific visual features to the decoder, ensuring that the mask generated is physically anchored to the reasoning context.

Interleaving Conversation and Grounding

Beyond simple queries, researchers are moving toward Grounded Conversation Generation (GCG). In this paradigm, a model doesn't just output a mask at the end of a sentence; it interleaves masks directly into a natural dialogue.

Hanoona Rasheed and the team behind GLaMM enabled models to ground specific phrases rather than entire sentences. For example, if a model says, "The [man] wearing a [red hat] is holding a [camera]," it can generate three distinct <SEG> tokens, each triggering a mask for the corresponding bracketed entity. This creates a seamless multimodal output where the text provides the "what" and "why," while the masks provide the "where."

To handle the "messy" side of these conversations—such as when a user asks to segment an object that isn't there—GSVA introduced a [REJ] (reject) token. This allows the model to explicitly signal that a reasoning path has led to a null set, preventing the "hallucinated" masks that plague simpler systems.

Extending to Time and Space

The reasoning segmentation framework is rapidly expanding into more complex dimensions, specifically video and 3D environments.

  1. Temporal Reasoning: In video, the model must track an entity through time based on intent. Zechen Bai and others in One Token to Seg Them All show that the same token-based mechanism can trigger spatio-temporal decoders. This allows a model to answer questions like "segment the person who tripped" by identifying the action in the temporal stream and grounding the actor across multiple frames.
  2. 3D and Part-Level Reasoning: Models like PARIS3D by Amrin Kareem apply this to 3D point clouds, segmenting functional parts (e.g., "the part used to pour water") rather than whole objects.

Specialized Domain Reasoning

The power of using an LLM as the reasoning engine is most apparent in specialized fields where "common sense" is actually "expert knowledge."

  • Medical Imaging: In MedSeg-R, Yu Huang shows how a model can interpret a clinical question ("What is causing the patient's symptoms?") and simultaneously provide a textual diagnosis and a pixel-level mask of the pathology.
  • Remote Sensing: LISAT, developed by Tsung-Han Wu, applies reasoning to satellite imagery, allowing users to find objects based on complex spatial descriptions (e.g., "the light-colored truck parked diagonally") that traditional category-based models would miss.

By treating segmentation as a linguistic output triggered by latent reasoning, these models move closer to a human-like understanding of the visual world—one where seeing is not just about recognizing shapes, but about understanding their context and purpose.

Papers in this version

Showing 60 of 149 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reasoning_segmentation_with_multimodal_l_3ce17cc3,
  title = {Reasoning Segmentation with Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-segmentation-with-multimodal-large-language-models/rdr_3ce17cc377304cfeb95bda146fee56d7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_reasoning_segmentation_with_multimodal_l_3ce17cc3_v1,
  title = {Reasoning Segmentation with Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-segmentation-with-multimodal-large-language-models/rdr_3ce17cc377304cfeb95bda146fee56d7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}