Abstract
Existing works on reasoning segmentation either connect hidden features from a language model directly to a mask decoder or represent positions in text, which limits interpretability and semantic detail. To solve this, we present CoPRS, a Multi-modal Chain-of-Thought (MCoT)-based positional perception model that bridges language reasoning to segmentation through a differentiable and interpretable positional prior instantiated as a heatmap. By making the reasoning process clear via MCoT and expressing it as a dense, differentiable heatmap, this interface enhances interpretability and diagnostic analysis and yields more concentrated evidence on the target. A learnable concentration token aggregates features of the image and reasoning text to generate this positional prior, which is decoded to precise masks through a lightweight decoder, providing a direct connection between reasoning and segmentation. Across the RefCOCO series and ReasonSeg, CoPRS matches or surpasses the best reported metrics on each standard split under comparable protocols. Extensive experiments demonstrate a strong positive correlation among the CoT trajectory, the generated heatmap, and the decoded mask, supporting an interpretable alignment between the reasoning output and downstream mask generation.
Introduction: Bridging the Gap in Reasoning Segmentation
Reasoning segmentation represents a significant challenge in visual grounding. Unlike standard referring expression segmentation, which identifies objects based on direct descriptions, reasoning segmentation requires the model to interpret complex, multi-step instructions—for example, identifying "the animal that is trailing the quadcopter and partially occluded by trees."
Current methodologies typically fall into two categories:
- Latent Reasoning: Models like LISA decode masks directly from hidden language features. While effective, these "black box" approaches lack interpretability, making it difficult to understand the model's decision-making process.
- Text-Based Reasoning: Models like Seg-Zero output discrete coordinates in text. While explicit, this approach is brittle; small formatting errors or single-point coordinates often fail to capture the nuanced shapes and fine-grained semantics of complex objects.
CoPRS introduces a "middle way" by utilizing a differentiable heatmap—a positional prior—that is both human-interpretable and representatively rich.
Methodology: MCoT and the Positional Prior
The CoPRS architecture is a one-stage, end-to-end system built on a Multi-modal Large Language Model (MLLM), specifically Qwen2.5-VL. The process begins with the generation of a Multi-modal Chain-of-Thought (MCoT) explanation, which concludes with a specialized <REF_POS> token.
The Heatmap Interface
The hidden state of the <REF_POS> token serves as a "concentration query." This query interacts with image features from a high-resolution vision backbone (ViT-H) through multi-head attention to generate a dense heatmap. This heatmap acts as a positional prior, visualizing the spatial focus derived from the model's reasoning. A lightweight decoder (4.7M parameters) then refines this prior into a high-resolution binary mask.
Unified Training with GRPO
To optimize both reasoning and segmentation, CoPRS employs Group Relative Policy Optimization (GRPO). This reinforcement learning strategy, popularized by the DeepSeek-R1 series, allows the model to explore multiple reasoning paths and receive rewards based on the accuracy of the resulting segmentation. The training objective integrates this RL approach with standard supervised losses, including BCE, Dice, and Focal loss.
Experimental Results
CoPRS was evaluated on standard benchmarks including RefCOCO, RefCOCO+, RefCOCOg, and the more complex ReasonSeg dataset.
- State-of-the-Art Performance: CoPRS-7B consistently matches or exceeds the performance of larger models. On RefCOCO+ testA, it achieved 80.3 cIoU. In zero-shot evaluations on ReasonSeg, it reached 65.2 gIoU, significantly outperforming previous leaders like LISA-13B (57.7).
- Interpretability and Correlation: A key finding of the study is the strong positive correlation between the quality of the Chain-of-Thought reasoning, the concentration of the heatmap, and the final mask accuracy. This confirms that improved "thinking" directly translates to more precise spatial grounding.
- Efficiency: The addition of the heatmap interface and lightweight decoder introduces negligible computational overhead compared to standard MLLM baselines.
Conclusion and Future Work
CoPRS demonstrates that a differentiable heatmap interface can effectively bridge the gap between linguistic reasoning and visual segmentation. By providing an interpretable spatial prior, the model allows for better diagnostic analysis and superior performance on complex reasoning tasks.
Future research will address current limitations, such as the resolution bottleneck for very small objects and ambiguity in dense scenes, by exploring multi-scale feature integration and tighter coupling between visual and linguistic features.
Summary updated