Introduction
In modern computer vision, semantic segmentation often relies on a two-stage pipeline: an initial model generates a "coarse" mask, which is then sharpened by a refinement algorithm, such as the Segment Anything Model (SAM). However, refinement is not always beneficial; it can introduce over-segmentation or noise. A critical bottleneck in deploying these systems is the lack of ground-truth labels during inference, making it impossible to determine if a refinement step actually improved the mask. This paper introduces Judging From Support-set (JFS), a framework that repurposes Few-Shot Segmentation (FSS) models to evaluate mask quality without requiring target labels.
Methodology: The JFS Framework
The core innovation of JFS is the inversion of standard Few-Shot Segmentation logic. In a typical FSS task, a model uses a high-quality "support mask" to find an object in a new "query image." JFS flips this: it treats the refined mask (whose quality is unknown) as the support mask. It then asks an FSS model (specifically SegGPT) to use this mask to segment a separate image where the ground truth is already known.
The underlying logic is that a high-quality refined mask will serve as a better prompt for the FSS model, leading to a more accurate prediction on the known image. By comparing the error rates ($E_c$ for coarse and $E_r$ for refined), the system can automatically decide whether the refinement was a success or a failure, choosing the mask that is more representative of the object class.
Experimental Results
The researchers validated JFS using the PASCAL VOC 2012 dataset and Wikimedia Commons images. The pipeline utilized Class Activation Maps (CAM) for initial coarse masks and SAM Enhanced Pseudo-Labels (SEPL) for refinement.
Key quantitative findings include:
- Recovery from Failure: In cases where refinement significantly degraded mask quality (the "Low 20" group), SEPL dropped the mIoU to 33.2. JFS successfully identified these failures, opting for the better coarse alternatives and raising the mIoU back to 50.5.
- Overall Performance: On a random sample of 400 images, JFS achieved an mIoU of 65.2, outperforming both the coarse (59.8) and refined (64.5) baselines.
- Success Rate: JFS correctly identified the superior mask with a success rate of up to 68.4% in high-improvement scenarios.
Discussion and Limitations
JFS represents a model-agnostic approach to segmentation monitoring, treating the refinement process as a black box. However, the authors identify several challenges for future work. If the coarse and refined masks cover identical "key features," the FSS judge may produce indistinguishable results. Additionally, background interference in either the query or support images can lead to false positives. Future iterations may involve fine-tuning FSS models specifically for the "judging" task rather than general segmentation to further improve reliability.
Summary updated