INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

Jian Hu, Zixu Cheng, Shaogang Gong

Abstract

Task-generic promptable segmentation aims to identify and segment objects based on high-level linguistic descriptions (e.g., "find the polyp"). However, Vision-Language Models (VLMs) often suffer from hallucinations in complex scenes, leading to inaccurate prompts for downstream segmentation models. We present INT (Instance-specific Negative Mining), a training-free, test-time adaptation method that refines instance-specific prompts by measuring VLM confidence drops upon object occlusion. By iteratively "mining" its own errors, INT outperforms several supervised benchmarks on challenging camouflaged and medical datasets.

Introduction

Traditional image segmentation relies heavily on large-scale manual labeling. While models like the Segment Anything Model (SAM) have introduced "promptable" segmentation, they typically require per-image manual intervention such as points or boxes. Task-generic segmentation seeks to automate this by using a single high-level prompt for an entire dataset.

The primary challenge in this paradigm is the reliability of the VLM. In complex scenarios—such as a frog camouflaged against tree bark—VLMs may misidentify background elements as target objects. Without ground-truth labels at test time, these errors propagate, resulting in poor segmentation masks. INT addresses this by using the model's own sensitivity to occlusion as a self-correction mechanism.

Methodology: Instance-Specific Negative Mining

INT treats the VLM's internal consistency as a proxy for ground truth. The pipeline operates in two primary stages:

1. Negative Mining for Prompt Refinement

The core innovation of INT is using image inpainting to verify VLM predictions.

  • Candidate Generation: A VLM identifies potential objects within image patches.
  • Negative Mining: An inpainting module (e.g., Stable Diffusion) "erases" the suspected object. The system then compares the VLM's confidence score before and after erasure.
  • The Logic: A significant drop in confidence indicates a high-probability match, whereas a minimal change suggests the VLM was attending to irrelevant background features (a "hard false" negative).
The motivation behind INT: using occlusion to identify true positives by observing confidence drops in the VLM.

This process is repeated iteratively. By cumulatively multiplying these "difference scores" over several epochs, the model amplifies consistent detections and suppresses sporadic hallucinations.

2. Semantic Mask Generation

Refined prompts are passed to GroundingDINO to generate bounding boxes, which then guide SAM to produce precise masks. A final CLIP-based semantic similarity check filters out residual noise, ensuring the mask aligns with the original task description.

Experiments and Results

INT was evaluated on six datasets across two domains: Camouflaged Object Detection (COD10K, CAMO, CHAMELEON) and Medical Image Segmentation (CVC-ColonDB, Kvasir-SEG, ISIC).

Performance Benchmarks

INT demonstrated superior performance, often exceeding supervised methods:

  • COD10K: INT achieved a Mean Absolute Error (M) of 0.037, surpassing scribble-supervised (0.038) and point-supervised (0.039) models.
  • Medical Domain: On CVC-ColonDB, INT reached an E-measure (Eφ) of 0.589, significantly outperforming other task-generic methods like GenSAM (0.494).
Effect of negative mining on the cosine similarity between ground truth and predicted instance-specific prompts across different datasets.

The iterative refinement typically converges within five iterations, effectively "cleaning" the prompts without requiring domain-specific fine-tuning.

Discussion and Conclusion

INT represents a significant step toward autonomous, high-precision segmentation. By leveraging the internal consistency of frozen VLMs through occlusion, it achieves state-of-the-art results in a training-free manner.

However, the computational cost of running multiple models (VLM, Diffusion, GroundingDINO, SAM) iteratively is high, limiting real-time applicability. Future research may focus on distilling this negative mining logic into more efficient, single-pass architectures or extending the method to more abstract task descriptions.

Summary updated

Abstract

Task-generic promptable image segmentation aims to achieve segmentation of diverse samples under a single task description by utilizing only one task-generic prompt. Current methods leverage the generalization capabilities of Vision-Language Models (VLMs) to infer instance-specific prompts from these task-generic prompts in order to guide the segmentation process. However, when VLMs struggle to generalise to some image instances, predicting instance-specific prompts becomes poor. To solve this problem, we introduce \textbf{I}nstance-specific \textbf{N}egative Mining for \textbf{T}ask-Generic Promptable Segmentation (\textbf{INT}). The key idea of INT is to adaptively reduce the influence of irrelevant (negative) prior knowledge whilst to increase the use the most plausible prior knowledge, selected by negative mining with higher contrast, in order to optimise instance-specific prompts generation. Specifically, INT consists of two components: (1) instance-specific prompt generation, which progressively fliters out incorrect information in prompt generation; (2) semantic mask generation, which ensures each image instance segmentation matches correctly the semantics of the instance-specific prompts. INT is validated on six datasets, including camouflaged objects and medical images, demonstrating its effectiveness, robustness and scalability.
BibTeX
@inproceedings{hu2025int,
  author = {Jian Hu and Zixu Cheng and Shaogang Gong},
  title = {{INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/124},
}
Figure