Abstract
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentation and semantic correspondence are jointly guided by a structured teacher in addition to the bounding box supervision. The teacher is a structured energy model incorporating a pairwise potential and a cross-image potential to model the pairwise pixel relationships both within and across the boxes. Minimizing the teacher energy simultaneously yields refined object masks and dense correspondences between intra-class objects, which are taken as pseudo-labels to supervise the task network and provide positive/negative correspondence pairs for dense contrastive learning. We show a symbiotic relationship where the two tasks mutually benefit from each other. Our best model achieves 37.9% AP on COCO instance segmentation, surpassing prior weakly supervised methods and is competitive to supervised methods. We also obtain state of the art weakly supervised results on PASCAL VOC12 and PF-PASCAL with real-time inference.
Introduction
In computer vision, the cost of data annotation remains a significant barrier to scaling models for complex tasks. While bounding box labels are relatively inexpensive, pixel-level instance masks and semantic keypoint correspondences typically require extensive human effort. DiscoBox addresses this challenge by leveraging a symbiotic relationship between instance segmentation and semantic correspondence. By learning these tasks jointly under weak supervision from bounding boxes, the framework achieves high-quality results without the need for dense manual annotations.
Methodology: Self-Ensembling with a Structured Teacher
The core of DiscoBox is a self-ensembling architecture comprising a "Student" task network and a "Structured Teacher."
The Task Network (Student)
The student network is an object detector (e.g., YOLACT++ or SOLOv2) adapted to predict instance masks. It is initially supervised using Multiple Instance Learning (MIL), which utilizes the "tightness" of bounding boxes—the assumption that every row and column of a box must contain at least one foreground pixel—to generate coarse initial masks.
The Structured Teacher
The Structured Teacher refines the student's predictions by enforcing geometric and appearance consistency. It employs a structured energy model with three key potentials:
- Unary Potential: Derived from the student's current mask predictions.
- Pairwise Potential: Encourages local smoothness by ensuring neighboring pixels with similar appearances share the same label.
- Cross-Image Potential: Leverages a memory bank of object features to find correspondences across different images of the same class.
By solving an optimal transport problem, the teacher generates refined masks and dense correspondences. These serve as pseudo-labels to train the student via consistency, contrastive, and MIL losses.
Experimental Results
DiscoBox was evaluated on major benchmarks including COCO, PASCAL VOC 2012, and PF-PASCAL.
- Instance Segmentation: On COCO, DiscoBox achieved 37.9% AP, setting a new state-of-the-art for weakly supervised methods and even outperforming some fully supervised baselines like Mask R-CNN.
- Semantic Correspondence: On PF-PASCAL, it reached 59.3% [email protected], significantly exceeding previous weakly supervised approaches that lacked localization priors.
- Efficiency: The complex teacher model is only utilized during training. Consequently, the student network maintains real-time inference speeds, making it practical for deployment.
Conclusion
DiscoBox demonstrates that bounding box supervision, when combined with structured self-ensembling, is sufficient to learn high-quality instance masks and semantic correspondences. By bridging the gap between these two tasks, the framework provides a scalable solution for dense visual understanding. Future work may extend this approach to fully unsupervised object discovery and 3D vision tasks where pixel-level correspondence is vital.
Summary updated