Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting

Kai Ye, Xiaotong You, Jianghang Lin, Jiayi Ji, Pingyang Dai, Liujuan CaoarXiv 2025 · DOI · arXiv

Abstract

Reasoning segmentation requires models to interpret complex, context-dependent instructions to identify and segment specific objects. Current paradigms often rely on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL), which can lead to catastrophic forgetting or training instability. This paper introduces EVOL-SAM3, a novel training-free framework that conceptualizes reasoning segmentation as an inference-time evolutionary search. By maintaining a population of prompt hypotheses and refining them through iterative "Visual Arena" tournaments, EVOL-SAM3 allows frozen foundation models to self-correct and achieve superior localization. Experimental results on the ReasonSeg benchmark demonstrate that EVOL-SAM3 (7B) outperforms larger supervised models like LISA-13B, achieving state-of-the-art performance without any parameter updates.

Introduction

Traditional image segmentation typically relies on predefined labels (e.g., "dog," "car"). In contrast, Reasoning Segmentation involves interpreting complex instructions like "segment the thing the person uses to fish." Most current solutions utilize Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, SFT often results in domain dependency and the loss of general reasoning abilities, while RL is notoriously unstable.

Existing training-free methods often follow a single-pass "generate-then-segment" approach. If the model's initial hypothesis is incorrect, it lacks a mechanism for self-correction, frequently leading to hallucinations. EVOL-SAM3 addresses this by replacing static inference with a dynamic "Generate-Evaluate-Evolve" loop, leveraging the discriminative power of Multimodal Large Language Models (MLLMs).

Methodology

EVOL-SAM3 utilizes a frozen MLLM (e.g., Qwen2.5-VL) as the reasoning engine and SAM 3 as the execution module. The framework operates in three distinct phases:

  1. Semantic Meta-Planning (Initialization): An MLLM acts as a meta-generator, expanding a user's abstract query into a diverse population of visually grounded prompt hypotheses, including spatial details and clarifying attributes.
  2. Evolutionary Reasoning Loop:
    • Execution & Verification: SAM 3 generates masks for each prompt, which are then filtered by a "Semantic Gate" to ensure basic logical consistency.
  3. Visual Arena: The MLLM acts as a judge in pairwise tournaments, comparing masks to determine which better fits the query. This win-rate establishes a prompt's "fitness."
    • Semantic Mutation: Elite prompts are mutated to explore refined descriptions, correcting semantic or spatial errors.
  4. Final Arbitration (Heterogeneous Arena): To ensure robustness, the system compares the best text-evolved mask against a mask generated from the MLLM’s raw geometric intuition. A "double-blind swap" strategy is employed to eliminate positional bias in the MLLM judge.

Experimental Evaluation

The researchers evaluated EVOL-SAM3 on several key benchmarks, including ReasonSeg (for complex reasoning) and the RefCOCO family (for referring expression segmentation).

Key Results

EVOL-SAM3 achieves remarkable results without a single parameter update:

  • Outperforming Supervised Models: On the ReasonSeg benchmark, EVOL-SAM3 (7B) achieved 72.5 gIoU, significantly surpassing the popular supervised model LISA-13B (61.3 gIoU).
  • Efficiency through Evolution: While the "SAM 3 Agent" baseline requires a massive 72B parameter model to reach high performance (70.8 gIoU), EVOL-SAM3 achieves superior results using only a 7B parameter model.
  • Complex Query Handling: On the "Test (Long)" subset of ReasonSeg, which features the most intricate instructions, EVOL-SAM3 reached 74.3 gIoU, outperforming RSVP (GPT-4o) at 61.9 gIoU.

Discussion and Conclusion

A core insight of this work is that MLLMs are often stronger at discriminating between options than generating a perfect answer on the first attempt. By "mining" the latent potential of frozen models through inference-time optimization, EVOL-SAM3 avoids the pitfalls of weight updates.

While the evolutionary loop introduces higher inference latency compared to single-pass models, the gains in accuracy and robustness are substantial. Future work may focus on more efficient search strategies to reduce computational overhead or extending this evolutionary prompting paradigm to video-based reasoning tasks.

Summary updated

Abstract

Reasoning Segmentation requires models to interpret complex, context-dependent linguistic queries to achieve pixel-level localization. Current dominant approaches rely heavily on Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, SFT suffers from catastrophic forgetting and domain dependency, while RL is often hindered by training instability and rigid reliance on predefined reward functions. Although recent training-free methods circumvent these training burdens, they are fundamentally limited by a static inference paradigm. These methods typically rely on a single-pass "generate-then-segment" chain, which suffers from insufficient reasoning depth and lacks the capability to self-correct linguistic hallucinations or spatial misinterpretations. In this paper, we challenge these limitations and propose EVOL-SAM3, a novel zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process. Instead of relying on a fixed prompt, EVOL-SAM3 maintains a population of prompt hypotheses and iteratively refines them through a "Generate-Evaluate-Evolve" loop. We introduce a Visual Arena to assess prompt fitness via reference-free pairwise tournaments, and a Semantic Mutation operator to inject diversity and correct semantic errors. Furthermore, a Heterogeneous Arena module integrates geometric priors with semantic reasoning to ensure robust final selection. Extensive experiments demonstrate that EVOL-SAM3 not only substantially outperforms static baselines but also significantly surpasses fully supervised state-of-the-art methods on the challenging ReasonSeg benchmark in a zero-shot setting. The code is available at https://github.com/AHideoKuzeA/Evol-SAM3.
BibTeX
@misc{ye2025evolving,
  author = {Kai Ye and Xiaotong You and Jianghang Lin and Jiayi Ji and Pingyang Dai and Liujuan Cao},
  title = {{Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting}},
  year = {2025},
  eprint = {2512.24702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.24702},
}
Figure