Discriminatively Matched Part Tokens for Pointly Supervised Instance Segmentation
Lists
Only you see this
Abstract
The self-attention mechanism of vision transformer has demonstrated potential for instance segmentation even using a single point as supervision. However, when it comes to objects with significant deformation and variations in appearance, this attention mechanism encounters a challenge of semantic variation among object parts. In this study, we propose discriminatively matched part tokens (DMPT), to extend the capacity of self-attention for pointly supervised instance segmentation. DMPT first allocates a token for each object part by finding a semantic extreme point, and then introduces part classifiers with deformable constraint to re-estimate part tokens which are utilized to guide and enhance the fine-grained localization capability of the self-attention mechanism. Through iterative optimization, DMPT matches the most discriminative part tokens which facilitate capturing fine-grained semantics and activating full object extent. Extensive experiments on PASCAL VOC and MS-COCO segmentation datasets show that DMPT respectively improves the state-of-the-art method by 2.0% mAP50 and 1.6% AP, achieving the best performance under point supervision. DMPT is combination with the Segment Anything Model (SAM), demonstrating the great potential to reform point prompt learning. Code is enclosed in the supplementary material.
BibTeX
@article{guo2025discriminatively,
author = {Zonghao Guo and Mingxiang Liao and Zhiliang Peng and Yidan Zhang and Peng Yuan and Qixiang Ye and Fang Wan},
title = {{Discriminatively Matched Part Tokens for Pointly Supervised Instance Segmentation}},
journal = {Int. J. Comput. Vis.},
year = {2025},
doi = {10.1007/s11263-025-02547-8},
}Loading related content...