Competing for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation

Shaheer U. Saeed, Shiqi Huang, João Ramalhinho, Iani J. M. B. Gayo, Nina Montaña-Brown, Ester Bonmati, Stephen P. Pereira, Brian R Davidson, Dean C. Barratt, Matthew J. Clarkson, Yipeng Hu

Abstract

Weakly-supervised semantic segmentation (WSSS) methods, reliant on image-level labels indicating object presence, lack explicit correspondence between labels and regions of interest (ROIs), posing a significant challenge. Despite this, WSSS methods have attracted attention due to their much lower annotation costs compared to fully-supervised segmentation. Leveraging reinforcement learning (RL) self-play, we propose a novel WSSS method that gamifies image segmentation of a ROI. We formulate segmentation as a competition between two agents that compete to select ROI-containing patches until exhaustion of all such patches. The score at each time-step, used to compute the reward for agent training, represents likelihood of object presence within the selection, determined by an object presence detector pre-trained using only image-level binary classification labels of object presence. Additionally, we propose a game termination condition that can be called by either side upon exhaustion of all ROI-containing patches, followed by the selection of a final patch from each. Upon termination, the agent is incentivised if ROI-containing patches are exhausted or disincentivised if a ROI-containing patch is found by the competitor. This competitive setup ensures minimisation of over- or under-segmentation, a common problem with WSSS methods. Extensive experimentation across four datasets demonstrates significant performance improvements over recent state-of-the-art methods.
BibTeX
@article{saeed2025competing,
  author = {Shaheer U. Saeed and Shiqi Huang and João Ramalhinho and Iani J. M. B. Gayo and Nina Montaña-Brown and Ester Bonmati and Stephen P. Pereira and Brian R Davidson and Dean C. Barratt and Matthew J. Clarkson and Yipeng Hu},
  title = {{Competing for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2024.3474094},
}

Loading related content...

Figure