CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation

Zhenyu Lu, Liupeng Li, Jinpeng Wang, Yan Feng, Bin Chen, Ke Chen, Yaowei Wang

TL;DR

CoPRS introduces a novel paradigm for reasoning segmentation by bridging language reasoning and mask generation through an interpretable, dense, and differentiable positional heatmap. By leveraging Multimodal Chain-of-Thought (MCoT) and Group Relative Policy Optimization (GRPO), it achieves state-of-the-art performance across standard benchmarks while making the intermediate spatial localization process fully transparent and diagnostic.

What Problem Does This Paper Address?

Reasoning segmentation tasks require a model to outline an object in an image based on complex, free-form, and often implicit natural language instructions (e.g., "Segment the tool used to hit the ball in a tennis game").

Existing approaches generally fall into two flawed categories:

  1. Latent reasoning methods: These decode segmentation masks directly from the hidden states of a Large Language Model (LLM). While effective, they act as a "black box"—the intermediate spatial decisions are non-transparent and uncontrollable.
  2. Text-based reasoning methods: These prompt the LLM to output discrete textual coordinates (like bounding boxes or point coordinates) which are then passed to a downstream segmenter like SAM. While explicit, this interface is inflexible, misses fine-grained visual semantics, and is highly fragile to formatting errors or out-of-bounds coordinates.

CoPRS addresses this gap by creating a dense, differentiable, and human-interpretable intermediate interface: a positional prior heatmap that perfectly balances representational fidelity with interpretability.

Figure 1: Illustration of paradigms for reasoning segmentation. (a) is exemplified by LISA (Lai et al., 2024), and (b) by Seg-Zero (Liu et al., 2025). Our CoPRS (c) bridges MCoT reasoning to segmentation through a differentiable and interpretable positional prior.

Methods

CoPRS utilizes a unified, one-stage, end-to-end architecture consisting of the following core components:

  • MLLM Backbone & MCoT: The model uses Qwen2.5-VL as its core language engine. Given an image-instruction pair, it executes a Multimodal Chain-of-Thought (MCoT) inside a <think> block to reason about the target before outputting a learnable concentration token (<REF_POS>).
  • Positional Prior Heatmap: A query head maps the embedding of the concentration token into a query ($Q$). Multi-head attention is then computed between $Q$ and image keys ($K$) derived from a frozen vision backbone (ViT-H from SAM). This attention map is processed into a dense $256\times256$ heatmap that acts as the positional prior.
  • Lightweight Decoder: A small 4.7M parameter mask decoder uses a Two-Way Transformer to perform bidirectional cross-attention between the image features and the positional prior, refining the heatmap into a precise, high-resolution binary mask.
  • Unified Training Framework: The entire system is optimized in a single loop. It couples Group Relative Policy Optimization (GRPO) on the language path (rewarding both mask quality and CoT formatting compliance) with supervised segmentation losses (Binary Cross-Entropy on the heatmap, alongside Dice and Focal losses on the final mask) on the vision path.
Figure 2: Overall architecture. Given image and text inputs, the policy generates CoT and concen- tration tokens, which query image keys to generate a positional prior, that is then decoded to masks. The policy and segmentation modules are jointly optimized.

Datasets

The model was trained and evaluated on several prominent datasets:

  • RefCOCO, RefCOCO+, and RefCOCOg: Standard benchmarks for referring expression segmentation, evaluated using cumulative Intersection over Union (cIoU).
  • ReasonSeg: A dataset specifically designed for complex reasoning segmentation. CoPRS was evaluated on ReasonSeg in a zero-shot setting (without training on its images) using both cIoU and generalized IoU (gIoU) to test its true generalization capabilities.

Key Insights and Ideas

  • The Heatmap as a Universal Interface: Instead of forcing language models to act like coordinate generators or treating their hidden states as magic vectors, mapping a specific token to a dense attention heatmap allows the language model to naturally express spatial focus in a way that downstream vision decoders natively understand.
  • Reasoning Quality Directly Drives Performance: By using Gemini-2.5-Flash as an independent automatic evaluator to score the logical consistency of the generated CoT trajectories, the authors proved a strong positive correlation: better, more logical text reasoning directly yields higher-quality heatmaps and more accurate final masks.
  • RL and Supervised Synergy: The combination of GRPO and pixel-level supervision creates a powerful training dynamic. Reinforcement learning expands the model's capacity to explore and reason, while direct supervised segmentation losses ensure that this reasoning accurately sharpens the physical boundaries of the generated masks.

Key Results

  • State-of-the-Art Metrics: CoPRS matches or surpasses the best reported metrics across all standard splits of RefCOCO, RefCOCO+, and RefCOCOg. For example, CoPRS-7B achieves a cIoU of 85.3 on RefCOCO testA and 76.2 on RefCOCOg test.
  • Superior Zero-Shot Generalization: On the complex ReasonSeg dataset, CoPRS-7B achieves 65.2 gIoU on the validation set and 59.8 gIoU on the test set, noticeably outperforming previous latent and text-based reasoning methods.
  • Efficiency at Scale: Due to the efficiency of the positional prior interface, the smaller CoPRS-3B model frequently outperforms older 7B and 13B parameter baselines.

Limitations and Future Work

The authors identify two primary failure modes:

  • Resolution Bottleneck: Because the positional prior heatmap is computed at $256\times256$ and SAM embeddings sit at $64\times64$, extremely small or distant objects can "vanish" during downsampling, making them impossible for the decoder to reliably recover.
  • Same-Class Crowd Ambiguity: In dense groups of highly similar objects (e.g., a crowd of people), the positional prior sometimes spreads weakly across multiple candidates, showing that text-only instructions struggle to fully disambiguate near-duplicates at current resolutions.

Future Work: The authors suggest that incorporating multi-scale features, higher-resolution inputs, and tighter instance-level language grounding will be key to resolving these edge cases.

Summary updated

Abstract

Existing works on reasoning segmentation either connect hidden features from a language model directly to a mask decoder or represent positions in text, which limits interpretability and semantic detail. To solve this, we present CoPRS, a Multi-modal Chain-of-Thought (MCoT)-based positional perception model that bridges language reasoning to segmentation through a differentiable and interpretable positional prior instantiated as a heatmap. By making the reasoning process clear via MCoT and expressing it as a dense, differentiable heatmap, this interface enhances interpretability and diagnostic analysis and yields more concentrated evidence on the target. A learnable concentration token aggregates features of the image and reasoning text to generate this positional prior, which is decoded to precise masks through a lightweight decoder, providing a direct connection between reasoning and segmentation. Across the RefCOCO series and ReasonSeg, CoPRS matches or surpasses the best reported metrics on each standard split under comparable protocols, with performance at or above the prior state of the art across both validation and test partitions. Extensive experiments demonstrate a strong positive correlation among the CoT trajectory, the generated heatmap, and the decoded mask, supporting an interpretable alignment between the reasoning output and downstream mask generation. Collectively, these findings support the utility of this paradigm in bridging reasoning and segmentation and show advantages in concentration driven by reasoning and in more precise mask prediction. Code has been released at https://github.com/ZhenyuLU-Heliodore/CoPRS.
BibTeX
@inproceedings{lu2026coprs,
  author = {Zhenyu Lu and Liupeng Li and Jinpeng Wang and Yan Feng and Bin Chen and Ke Chen and Yaowei Wang},
  title = {{CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Fcsop01h40},
}
Figure