B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

Mario Markov, Stefan Maria Ailuro, Mohammad Mahdi, Luc Van Gool, Danda Pani PaudelarXiv 2026 · arXiv

TL;DR

B-GRTO introduces a mathematically grounded framework to jointly optimize a vision-language reasoning policy and a trainable, differentiable segmentation tool (like SAM3) using reinforcement learning. By cheaply bootstrapping the tool off-policy before joint training, it achieves faster convergence and state-of-the-art performance across challenging referring segmentation tasks without requiring domain-specific architectures.

What Problem Does This Paper Address?

Referring segmentation requires a model to generate a pixel-level mask based on an open-ended natural language prompt. This task becomes exceptionally difficult when the prompts are ambiguous (e.g., "heaven for sunbathers and swimmers") or the objects are hard to distinguish from their surroundings (e.g., camouflaged animals).

To tackle this, modern approaches pair a large Multimodal Vision-Language Model (MLLM) for reasoning with a dense segmentation decoder acting as a "tool." While reinforcement learning (RL) has successfully been used to improve the MLLM's reasoning capabilities, the segmentation tool is typically kept frozen during this process. This creates a critical bottleneck: a highly capable reasoning model remains limited by a tool that cannot adapt to subtle, domain-specific visual cues or the MLLM's unique prompting style.

Jointly optimizing both components is notoriously difficult due to an asymmetry in training dynamics: the MLLM policy is massive and expensive to train via RL action sampling, whereas the segmentation tool is lightweight and can be updated cheaply using differentiable objectives.

Examples of challenging referring segmentation tasks requiring both reasoning and tool tuning.

Methods

The authors propose Group Relative Tool Optimization (GRTO) and its two-stage variant, Bootstrapped-GRTO (B-GRTO), to unify reinforcement learning with differentiable tool fine-tuning:

  1. GRTO (Joint Optimization): Building on Group Relative Policy Optimization (GRPO), GRTO reuses the policy's sampled rollout groups to optimize the tool. Instead of relying solely on sparse policy rewards, it backpropagates a differentiable surrogate loss (combining Binary Cross-Entropy and soft-IoU) directly into the segmentation tool, weighting the gradients using importance sampling.
  2. B-GRTO (Two-Stage Framework): To address the training speed asymmetry, B-GRTO introduces an initial Bootstrapped Tool Optimization (BTO) stage. BTO uses a static experience replay buffer generated by the initial reference policy to pre-train and align the tool before entering the expensive joint RL phase. This mathematically anticipates the behavior of the post-trained policy, giving the tool a computational head start.

Throughout the experiments, the authors utilize InternVL3.5-8B as the reasoning policy backbone and SAM3 as the trainable segmentation tool.

Datasets

The framework was evaluated across three distinct and challenging referring segmentation domains:

  • Camouflage Detection: Evaluated on the widely used COD10K and CAMO benchmarks, with zero-shot generalization tested on NC4K.
  • Remote Sensing Segmentation: Evaluated on EarthReason, a large-scale geospatial pixel reasoning dataset featuring complex, multi-scale overhead imagery.
  • Challenging Reasoning Segmentation: Tested on ReasonSeg-X and ReasonSeg-R, benchmarks specifically designed to stress-test complex world-knowledge and multi-step reasoning.

Key Insights and Ideas

  • Tools Shouldn't Be Passive: Fine-tuning the decoder is just as important as fine-tuning the reasoning model. Even if an MLLM correctly identifies an object, a domain-shifted or unaligned decoder will produce low-quality masks.
  • Coordinated Alignment Wins: Simply training the components sequentially (e.g., training the policy first, then fine-tuning the tool on deterministic outputs) yields inferior results compared to joint optimization. The policy and tool must learn complementary behaviors.
  • Cheap Bootstrapping Yields High Efficiency: Pre-aligning the tool via BTO adds negligible computational overhead (comparable to just a single GRPO epoch) but drastically stabilizes joint training and accelerates policy convergence.

Key Results

  • State-of-the-Art Performance: In camouflage detection, B-GRTO achieved state-of-the-art results on COD10K and NC4K, ranking in the top two across all standard metrics (S-measure, weighted F-measure, E-measure, and MAE), outperforming heavily tailored domain-specific architectures.
  • Significant Baselines Lifts: On the EarthReason remote sensing dataset, GRTO and B-GRTO significantly improved both generalized IoU (gIoU) and cumulative IoU (cIoU) over the plain GRPO baseline.
  • Superior Training Efficiency: B-GRTO demonstrated the best performance-to-budget trade-off, converging much faster than standard joint training while maintaining or exceeding final mask quality.

Limitations and Future Work

  • Overfitting Risks: On very small datasets (such as ReasonSeg-X, which contains only 240 training samples), joint optimization runs the risk of overfitting the decoder, which can degrade zero-shot generalization performance in later epochs.
  • Dependency on the Initial Policy: The bootstrapping phase relies entirely on the quality of the initial replay buffer. If the reference policy completely fails to generate relevant prompts for a specific class of objects, the tool receives no meaningful learning signal for those cases.
  • Hyperparameter Sensitivity: The system is highly sensitive to learning rate mismatches between the massive policy model and the lightweight tool, requiring careful calibration.

Future work highlights exploring more adaptive replay strategies, such as dynamically refreshing the bootstrapping buffer as the policy improves, or implementing curriculum learning for highly complex examples.

Summary updated

Abstract

Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis. For complex referring segmentation, recent methods pair large vision-language models with segmentation decoders: the former analyzes the image and prompt, while the latter predicts the target mask. Although reinforcement learning improves reasoning-intensive vision-language systems, trainable tools such as segmentation decoders are typically optimized separately with differentiable objectives, and the principled integration of such objectives into reinforcement learning remains underexplored. Thus, we introduce group relative tool optimization (GRTO), a mathematically grounded framework for jointly optimizing a policy with differentiable tool use. GRTO reuses group relative policy optimization (GRPO) rollouts to optimize the auxiliary tool objective, letting decoder gradients complement policy rewards. Further, we derive Bootstrapped-GRTO (B-GRTO), a pre-training method that cheaply bootstraps the tool, leading to faster convergence and superior performance. Across three challenging referring segmentation settings, B-GRTO results in substantial improvements over plain GRPO, matching or surpassing domain-specific state-of-the-art methods. This demonstrates the value of unifying reinforcement learning with differentiable auxiliary objectives for reasoning-intensive segmentation.
BibTeX
@misc{markov2026b,
  author = {Mario Markov and Stefan Maria Ailuro and Mohammad Mahdi and Luc Van Gool and Danda Pani Paudel},
  title = {{B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation}},
  year = {2026},
  eprint = {2605.23500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.23500},
}
Figure