CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation

Zhenyu Lu, Liupeng Li, Jinpeng Wang, Yan Feng, Bin Chen, Ke Chen, Yaowei Wang

Abstract

Existing works on reasoning segmentation either connect hidden features from a language model directly to a mask decoder or represent positions in text, which limits interpretability and semantic detail. To solve this, we present CoPRS, a Multi-modal Chain-of-Thought (MCoT)-based positional perception model that bridges language reasoning to segmentation through a differentiable and interpretable positional prior instantiated as a heatmap. By making the reasoning process clear via MCoT and expressing it as a dense, differentiable heatmap, this interface enhances interpretability and diagnostic analysis and yields more concentrated evidence on the target. A learnable concentration token aggregates features of the image and reasoning text to generate this positional prior, which is decoded to precise masks through a lightweight decoder, providing a direct connection between reasoning and segmentation. Across the RefCOCO series and ReasonSeg, CoPRS matches or surpasses the best reported metrics on each standard split under comparable protocols. Extensive experiments demonstrate a strong positive correlation among the CoT trajectory, the generated heatmap, and the decoded mask, supporting an interpretable alignment between the reasoning output and downstream mask generation.

Introduction: Bridging the Gap in Reasoning Segmentation

Reasoning segmentation represents a significant challenge in visual grounding. Unlike standard referring expression segmentation, which identifies objects based on direct descriptions, reasoning segmentation requires the model to interpret complex, multi-step instructions—for example, identifying "the animal that is trailing the quadcopter and partially occluded by trees."

Current methodologies typically fall into two categories:

  1. Latent Reasoning: Models like LISA decode masks directly from hidden language features. While effective, these "black box" approaches lack interpretability, making it difficult to understand the model's decision-making process.
  2. Text-Based Reasoning: Models like Seg-Zero output discrete coordinates in text. While explicit, this approach is brittle; small formatting errors or single-point coordinates often fail to capture the nuanced shapes and fine-grained semantics of complex objects.

CoPRS introduces a "middle way" by utilizing a differentiable heatmap—a positional prior—that is both human-interpretable and representatively rich.

Comparison of reasoning segmentation paradigms: (a) Latent Reasoning, (b) Text-Based Reasoning, and (c) the proposed MCoT-Based Positional Perception (CoPRS).

Methodology: MCoT and the Positional Prior

The CoPRS architecture is a one-stage, end-to-end system built on a Multi-modal Large Language Model (MLLM), specifically Qwen2.5-VL. The process begins with the generation of a Multi-modal Chain-of-Thought (MCoT) explanation, which concludes with a specialized <REF_POS> token.

The Heatmap Interface

The hidden state of the <REF_POS> token serves as a "concentration query." This query interacts with image features from a high-resolution vision backbone (ViT-H) through multi-head attention to generate a dense heatmap. This heatmap acts as a positional prior, visualizing the spatial focus derived from the model's reasoning. A lightweight decoder (4.7M parameters) then refines this prior into a high-resolution binary mask.

Unified Training with GRPO

To optimize both reasoning and segmentation, CoPRS employs Group Relative Policy Optimization (GRPO). This reinforcement learning strategy, popularized by the DeepSeek-R1 series, allows the model to explore multiple reasoning paths and receive rewards based on the accuracy of the resulting segmentation. The training objective integrates this RL approach with standard supervised losses, including BCE, Dice, and Focal loss.

The CoPRS architecture and training framework, illustrating the flow from MCoT reasoning to heatmap generation and GRPO-based optimization.

Experimental Results

CoPRS was evaluated on standard benchmarks including RefCOCO, RefCOCO+, RefCOCOg, and the more complex ReasonSeg dataset.

  • State-of-the-Art Performance: CoPRS-7B consistently matches or exceeds the performance of larger models. On RefCOCO+ testA, it achieved 80.3 cIoU. In zero-shot evaluations on ReasonSeg, it reached 65.2 gIoU, significantly outperforming previous leaders like LISA-13B (57.7).
  • Interpretability and Correlation: A key finding of the study is the strong positive correlation between the quality of the Chain-of-Thought reasoning, the concentration of the heatmap, and the final mask accuracy. This confirms that improved "thinking" directly translates to more precise spatial grounding.
  • Efficiency: The addition of the heatmap interface and lightweight decoder introduces negligible computational overhead compared to standard MLLM baselines.

Conclusion and Future Work

CoPRS demonstrates that a differentiable heatmap interface can effectively bridge the gap between linguistic reasoning and visual segmentation. By providing an interpretable spatial prior, the model allows for better diagnostic analysis and superior performance on complex reasoning tasks.

Future research will address current limitations, such as the resolution bottleneck for very small objects and ambiguity in dense scenes, by exploring multi-scale feature integration and tighter coupling between visual and linguistic features.

Summary updated

Abstract

Existing works on reasoning segmentation either connect hidden features from a language model directly to a mask decoder or represent positions in text, which limits interpretability and semantic detail. To solve this, we present CoPRS, a Multi-modal Chain-of-Thought (MCoT)-based positional perception model that bridges language reasoning to segmentation through a differentiable and interpretable positional prior instantiated as a heatmap. By making the reasoning process clear via MCoT and expressing it as a dense, differentiable heatmap, this interface enhances interpretability and diagnostic analysis and yields more concentrated evidence on the target. A learnable concentration token aggregates features of the image and reasoning text to generate this positional prior, which is decoded to precise masks through a lightweight decoder, providing a direct connection between reasoning and segmentation. Across the RefCOCO series and ReasonSeg, CoPRS matches or surpasses the best reported metrics on each standard split under comparable protocols, with performance at or above the prior state of the art across both validation and test partitions. Extensive experiments demonstrate a strong positive correlation among the CoT trajectory, the generated heatmap, and the decoded mask, supporting an interpretable alignment between the reasoning output and downstream mask generation. Collectively, these findings support the utility of this paradigm in bridging reasoning and segmentation and show advantages in concentration driven by reasoning and in more precise mask prediction. Code, checkpoints and logs are released at https://github.com/ZhenyuLU-Heliodore/CoPRS.git.
BibTeX
@inproceedings{lu2026coprs,
  author = {Zhenyu Lu and Liupeng Li and Jinpeng Wang and Yan Feng and Bin Chen and Ke Chen and Yaowei Wang},
  title = {{CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Fcsop01h40},
}
Figure