Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation

Jian Hu, Jiayi Lin, Junchi Yan, Shaogang Gong

TL;DR

ProMaC (Prompt-Mask Cycle generation) is a training-free, inference-time framework that turns a common flaw in Multimodal Large Language Models (MLLMs)—hallucinations—into a powerful feature for task-generic promptable segmentation. By intentionally triggering and then systematically verifying MLLM hallucinations through an iterative prompt-mask feedback loop, ProMaC achieves state-of-the-art segmentation performance using only a single high-level task prompt.

What Problem Does This Paper Address?

Promptable segmentation models, like Meta's Segment Anything Model (SAM), excel at delineating object boundaries but lack the high-level semantic understanding required to know what to segment. As a result, they typically rely on manual, instance-specific prompts (like bounding boxes or points) for every single image.

To scale this up, researchers introduced task-generic promptable segmentation, which uses a single high-level prompt (e.g., "camouflaged animal") for an entire dataset. Current state-of-the-art methods use MLLMs to reason out specific object details from these generic prompts. However, MLLMs frequently suffer from hallucinations due to object co-occurrence priors—for example, falsely predicting a "leopard" simply because the image contains yellowish grass, or misidentifying a heavily camouflaged caterpillar as a leaf.

Figure 1: (a) During MLLM pretraining, leopards often co-occur with grass. If the lion is masked, the model incorrectly identifies it as a leopard based on the grass. (b) Directly inputting the image into MLLM causes the hidden caterpillar being incorrectly predicted as a leaf. Splitting the image results in interested objects being incomplete or absent, prompting MLLM to induce hallucinations and utilize prior knowledge to predict potential task-related objects within the image. Our visual contrastive reasoning then eliminates the hallucinations and validates the gathered predictions, aiding in the accurate identification of the caterpillar.

While conventional wisdom dictates that hallucinations must be completely eradicated, this paper proposes a counter-intuitive twist: MLLM hallucinations represent a rich, untapped source of pre-trained contextual knowledge that can be leveraged to enhance scene understanding if properly managed and verified.

Methods

The authors introduce ProMaC, a training-free, test-time adaptation framework that optimizes both prompts and segmentation masks through a cyclic interaction loop.

Figure 2: An overview of ProMac: Masks created iteratively by the mask generator guide the prompt generator to jointly improve instance-specific prompts and visual masking in segmentation.

The architecture consists of two primary components:

  1. Prompt Generator (Multi-scale Chain of Thought & Visual Contrastive Reasoning):
  2. Hypothesize: The input image is divided into multi-scale patches. This partial visibility intentionally triggers the MLLM's hallucinatory tendencies, forcing it to generate diverse hypotheses about potential objects based on its prior knowledge.
  3. Verify: To filter out incorrect co-occurrence hallucinations, ProMaC introduces Visual Contrastive Reasoning (VCR). It uses a generative inpainting model (Stable Diffusion) to remove the hypothesized object area from the image. By comparing the MLLM's output distribution on the original image versus this "background-only" contrastive image, the system effectively cancels out false positives caused by background context.
  4. Mask Generator (Mask Semantic Alignment):
  5. The refined instance-specific prompts are passed to SAM to generate detailed masks.
  6. Since SAM lacks semantic understanding, ProMaC uses Mask Semantic Alignment via CLIP to score the generated masks against the text prompts, ensuring the final mask aligns tightly with the intended task semantics.

This process runs iteratively: the masks generated in iteration $t$ serve as visual guidance to help the prompt generator focus on task-relevant regions in iteration $t+1$, progressively reducing irrelevant hallucinations.

Datasets

ProMaC was comprehensively evaluated across 5 distinct segmentation tasks spanning 12 diverse datasets:

  • Camouflaged Object Detection (COD): CHAMELEON, CAMO, and COD10K.
  • Medical Image Segmentation (MIS): CVC-ColonDB and Kvasir (for polyp segmentation), and ISIC (for skin lesion segmentation).
  • Transparent Object Detection (TOD): GSD and Trans10K-hard.
  • Open-Vocabulary Segmentation (OVS): PASCAL VOC, Pascal Context, and COCO-Object.
  • Spatial Reasoning (SR): What's Up dataset (used specifically to evaluate the VCR component).

Key Insights and Ideas

  • Embracing Hallucinations as Exploration: Instead of treating hallucinations as pure errors, ProMaC treats them as an exploratory mechanism. In complex or camouflaged scenes where direct visual cues are weak, hallucinations allow the model to "guess" plausible objects using broad contextual associations, which are later verified.
  • Visual Classifier-Free Guidance: The Visual Contrastive Reasoning (VCR) mechanism elegantly ports the concept of classifier-free guidance into visual prompt verification. Physically removing a hypothesized object via inpainting provides a clean, training-free way to measure an MLLM's true confidence in a visual region.
  • True Test-Time Adaptation: ProMaC requires zero fine-tuning or proprietary APIs (like GPT-4V). It works entirely at inference time using open-source models like LLaVA-1.5-13B and standard SAM.

Key Results

  • Camouflaged Object Detection: ProMaC set a new state-of-the-art for task-generic settings. On the COD10K dataset, it achieved a Mean Absolute Error ($M$) of 0.042, outperforming the previous best task-generic method, GenSAM ($M=0.058$), and even outperforming point-supervised methods.
  • Medical Image Segmentation: On the challenging CVC-ColonDB dataset, ProMaC reduced the Mean Absolute Error to 0.176, a massive improvement over GenSAM ($M=0.244$) and standard LLaVA+SAM ($M=0.491$).
  • Open-Vocabulary Segmentation: ProMaC achieved a 59.3% mIoU on PASCAL VOC, comfortably outperforming specialized open-vocabulary models like OVSegmentor (53.8%) without any task-specific training.

Limitations and Future Work

  • Inference Computational Cost: Because ProMaC relies on an iterative feedback loop involving multi-scale MLLM prompting, Stable Diffusion inpainting, and SAM mask generation over multiple epochs (typically 4 to 6 iterations per image), it introduces significant computational overhead at test time.
  • Cascading Dependencies: The framework's performance is tightly coupled with the capabilities of its sub-components. If the open-source MLLM fails to generate the correct initial hypothesis, or if the inpainting model introduces severe artifacts, the visual contrastive loop can break down. Future work could explore lighter-weight verification mechanisms to reduce test-time latency.

Summary updated

Abstract

Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting. While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in https://lwpyh.github.io/ProMaC/.
BibTeX
@inproceedings{hu2024leveraging,
  author = {Jian Hu and Jiayi Lin and Junchi Yan and Shaogang Gong},
  title = {{Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=9GhSOp1LYH},
}
Figure