Introduction
Image segmentation is a cornerstone of computer vision, yet its reliance on dense, pixel-level annotations presents a significant bottleneck, particularly in domains like medical imaging where expert time is at a premium. Weakly-supervised segmentation (WSS) addresses this by training models using only image-level labels (e.g., "this image contains a tumor"). However, WSS faces a persistent "correspondence gap": models often struggle to distinguish between the most discriminative parts of an object (leading to under-segmentation) and the surrounding background (leading to over-segmentation).
This paper introduces RL Self-Play (RLSP), a novel reinforcement learning framework that reframes segmentation as a competitive game. By tasking two agents with "claiming" object-related patches, the system learns to precisely delineate regions of interest (ROIs) without the need for manual masks.
The RLSP Framework
The core of the RLSP approach is the gamification of the segmentation process. The image is divided into a grid of patches, and two agents (Agent A and Agent B) compete to select patches that contain the target object.
1. The Competitive Mechanism
Agents take turns selecting patches. Once a patch is chosen, it is removed from the available pool. A pre-trained "referee"—a binary classifier trained on image-level labels—provides a reward signal by scoring the likelihood that a selected patch contains the object. The agent that selects the higher-scoring patch in a turn wins that round. This competition acts as a natural regularizer: if one agent fails to identify a relevant part of the object, the opponent will claim it to gain a reward, forcing both agents to be more thorough.
2. Termination and the "Showdown"
To ensure the agents are exhaustive, the game includes a strategic termination condition. An agent can choose to end the game at any time. However, if the opponent can still find a patch with a high object-presence score after the game is called, the agent who terminated the game receives a penalty. This "showdown" mechanism forces the agents to learn an optimal stopping strategy, effectively discovering the object's boundaries without hand-tuned probability thresholds.
3. Robust Training via Self-Play
To prevent the models from falling into local optima or cyclic behaviors, the authors employ Prioritized Fictitious Self-Play. Instead of simply playing against the most recent version of itself, the current agent competes against a distribution of its past iterations. This ensures the learned strategies are robust and generalize well across different image contexts.
Experimental Results
The RLSP method was evaluated across four diverse datasets, spanning standard computer vision benchmarks and complex medical imaging tasks:
- PASCAL VOC 2012 & MS COCO 2014: On PASCAL VOC, the model achieved 78.9 mIoU, surpassing the previous state-of-the-art (MARS) by 1.2 points.
- Medical Imaging (CMPS & LiTS): In the Male Pelvic Structures (CMPS) dataset, RLSP reached a 58.8 mDSC, significantly outperforming prior weakly-supervised methods (54.2 mDSC).
Qualitatively, RLSP demonstrates a superior ability to capture the full extent of objects. While traditional Class Activation Map (CAM) methods often highlight only the most distinct features (such as the head of an animal), RLSP successfully delineates the entire body by treating every unclaimed object patch as a potential reward for the competing agent.
Discussion and Limitations
RLSP offers a significant shift from traditional WSS by replacing static thresholding with a dynamic, learned search process. However, the method's performance is tied to two main factors:
- Patch Size: The resolution of the segmentation is limited by the grid size. Very small patches may lack the context needed for the detector to identify the object, while very large patches result in coarse boundaries.
- Detector Quality: The agents are only as effective as the "referee." Biases in the initial image-level classifier can propagate through the RL training process.
Future work aims to address these by exploring end-to-end training of the detector and agents simultaneously, as well as extending the competitive discovery framework to 3D localization and anomaly detection tasks.
Summary updated