From Sight to Insight: Enhancing Confusable Structure Segmentation via Vision-Language Mutual Prompting

Yixiang Huang, Yihao Zuo, Mengqiu Xu, Kaixin Chen, Ming Wu, Chuang Zhang, Zhanyu Ma, Jun Guo

Abstract

Confusable structure segmentation (CSS) is a type of semantic segmentation applied in remote sensing sea fog detection, medical image segmentation, camouflaged object detection, etc. Structural similarity and visual ambiguity are two critical issues in CSS that pose difficulties in distinguishing foreground objects from the background. Current methods focus primarily on enhancing visual representations and do not often incorporate multimodal information, which leads to performance bottlenecks. Inspired by recent achievements in vision-language models, we propose Vision-Language Mutual Prompting (VLMP), a novel and unified language-guided framework that leverages text prompts to enhance CSS. Specifically, VLMP consists of vision-to-language prompting and language-to-vision prompting, which bidirectionally model the interactions between visual and linguistic features, thereby facilitating cross-modal complementary information flow. To prevent the predominance of one modality over another, we design a feature integration modulator that modulates and balances feature weights for adaptive multimodal fusion. Our framework is designed to be modular and flexible, allowing for integration with any backbone, including CNNs and transformers. We evaluate VLMP with three diverse datasets: SFDD-H8, QaTa-COV19, and CAMO-COD10K. Extensive experiments demonstrate the effectiveness and superiority of the proposed framework over those of state-of-the-art methods across these datasets. This shift from basic sight to deeper insight in CSS through vision-language integration represents a significant advancement in the field.
BibTeX
@article{huang2026from,
  author = {Yixiang Huang and Yihao Zuo and Mengqiu Xu and Kaixin Chen and Ming Wu and Chuang Zhang and Zhanyu Ma and Jun Guo},
  title = {{From Sight to Insight: Enhancing Confusable Structure Segmentation via Vision-Language Mutual Prompting}},
  journal = {IEEE Trans. Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2025.3639888},
}

Loading related content...

Figure