INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

Jian Hu, Zixu Cheng, Shaogang Gong

Abstract

Task-generic promptable segmentation aims to segment diverse images using a single, broad text prompt. However, current systems struggle when Vision-Language Models (VLMs) misidentify objects in complex scenes. This paper introduces INT, a training-free method that utilizes "progressive negative mining" to iteratively test and refine VLM predictions. By masking out candidate objects and measuring the resulting drop in model confidence, INT accurately deduces instance-specific prompts without requiring ground-truth labels. This approach significantly improves segmentation performance on challenging tasks, including camouflaged object detection and medical image analysis.

Introduction

Foundation models like the Segment Anything Model (SAM) have revolutionized computer vision but typically rely on manual, per-image prompts such as points or bounding boxes. "Task-generic" segmentation seeks to eliminate this manual bottleneck by using a single broad prompt (e.g., "find the camouflaged animal") for an entire dataset.

To bridge the gap between generic prompts and specific image content, systems employ Vision-Language Models (VLMs) to generate "instance-specific" prompts (e.g., identifying that the animal in a specific image is a "frog"). However, VLMs frequently hallucinate or fail in ambiguous scenes. Without ground-truth labels at test time, identifying incorrect prompts is difficult, leading to error propagation. INT addresses this by treating the VLM as a black-box scorer to verify its own predictions through visual subtraction.

Methodology

INT (Instance-specific Negative Mining for Task-Generic Promptable Segmentation) is a training-free, test-time adaptation framework consisting of two primary stages:

1. Instance-Specific Prompt Generation via Negative Mining

INT first splits the input image into patches to force the VLM (LLaVA-1.5) to rely on local features, generating a set of candidate objects. To verify these guesses, INT employs an inpainting module (Stable Diffusion) to erase the predicted object from the scene. The modified image is fed back to the VLM; a significant drop in confidence for that object category indicates a correct identification. Through progressive negative mining, these confidence drops are accumulated across multiple iterations to filter out noise and lock onto the true target.

2. Semantic Mask Generation

Once the instance-specific prompt (e.g., "frog") is confirmed, GroundingDINO generates bounding boxes, which are then processed by SAM to create high-fidelity masks. A CLIP-based verification step ensures the final mask aligns semantically with the text prompt. This mask is also used to suppress background noise in subsequent iterations, further refining the model's focus.

The INT pipeline: (Top Left) Image splitting and initial VLM guessing. (Bottom) Progressive negative mining using inpainting to verify prompts. (Top Right) Final mask generation using GroundingDINO, SAM, and CLIP.

Key Results and Evaluation

The paper evaluates INT across six datasets in two challenging domains: Camouflaged Object Detection (COD) and Medical Image Segmentation (MIS).

  • Camouflaged Object Detection: On the CHAMELEON dataset, INT achieved a Mean Absolute Error (M) of 0.039 and an F-measure of 0.801, outperforming the previous state-of-the-art task-generic method, ProMaC.
  • Medical Image Segmentation: Despite the lack of domain-specific training for the underlying VLMs, INT successfully adapted to medical tasks. On the CVC-ColonDB dataset, INT achieved an M of 0.172, vastly outperforming direct VLM applications like GPT-4V+SAM (0.578).
Qualitative comparison of INT against Ground Truth (GT) and baseline methods (GenSAM, ProMaC) across camouflaged object detection and medical image segmentation tasks.

Discussion and Insights

The core innovation of INT is the realization that ground truth is not required to verify a VLM's prediction. By observing how a VLM reacts to the artificial removal of an object, the system can measure the accuracy of its own prompts. Progressive accumulation of these signals across iterations allows the model to overcome the inherent noise and hallucinations of standard VLMs.

While highly effective, the pipeline's reliance on multiple heavy models (LLaVA, Stable Diffusion, SAM) at test time introduces significant computational overhead. Future work may focus on optimizing this iterative process for real-time applications or enhancing the zero-shot domain knowledge of the underlying VLMs.

Conclusion

INT demonstrates that instance-specific negative mining is a powerful tool for task-generic segmentation. By turning the VLM's sensitivity to visual features into a self-verification mechanism, INT bridges the gap between broad task descriptions and precise instance-level masks, setting a new benchmark for training-free promptable segmentation.

Summary updated

BibTeX
@inproceedings{hu2025int,
  author = {Jian Hu and Zixu Cheng and Shaogang Gong},
  title = {{INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/124},
}
Figure