DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

Shiyi Lan, Zhiding Yu, Christopher Bongsoo Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S. Davis, Anima AnandkumarICCV 2021 · OpenReview

Abstract

We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentation and semantic correspondence are jointly guided by a structured teacher in addition to the bounding box supervision. The teacher is a structured energy model incorporating a pairwise potential and a cross-image potential to model the pairwise pixel relationships both within and across the boxes. Minimizing the teacher energy simultaneously yields refined object masks and dense correspondences between intra-class objects, which are taken as pseudo-labels to supervise the task network and provide positive/negative correspondence pairs for dense contrastive learning. We show a symbiotic relationship where the two tasks mutually benefit from each other. Our best model achieves 37.9% AP on COCO instance segmentation, surpassing prior weakly supervised methods and is competitive to supervised methods. We also obtain state of the art weakly supervised results on PASCAL VOC12 and PF-PASCAL with real-time inference.

Introduction

In computer vision, the cost of data annotation remains a significant barrier to scaling models for complex tasks. While bounding box labels are relatively inexpensive, pixel-level instance masks and semantic keypoint correspondences typically require extensive human effort. DiscoBox addresses this challenge by leveraging a symbiotic relationship between instance segmentation and semantic correspondence. By learning these tasks jointly under weak supervision from bounding boxes, the framework achieves high-quality results without the need for dense manual annotations.

DiscoBox jointly learns instance segmentation and semantic correspondence from bounding box supervision.

Methodology: Self-Ensembling with a Structured Teacher

The core of DiscoBox is a self-ensembling architecture comprising a "Student" task network and a "Structured Teacher."

The Task Network (Student)

The student network is an object detector (e.g., YOLACT++ or SOLOv2) adapted to predict instance masks. It is initially supervised using Multiple Instance Learning (MIL), which utilizes the "tightness" of bounding boxes—the assumption that every row and column of a box must contain at least one foreground pixel—to generate coarse initial masks.

The Structured Teacher

The Structured Teacher refines the student's predictions by enforcing geometric and appearance consistency. It employs a structured energy model with three key potentials:

  • Unary Potential: Derived from the student's current mask predictions.
  • Pairwise Potential: Encourages local smoothness by ensuring neighboring pixels with similar appearances share the same label.
  • Cross-Image Potential: Leverages a memory bank of object features to find correspondences across different images of the same class.
The DiscoBox architecture featuring the FPN-based student and the Structured Teacher for pseudo-label generation.

By solving an optimal transport problem, the teacher generates refined masks and dense correspondences. These serve as pseudo-labels to train the student via consistency, contrastive, and MIL losses.

Experimental Results

DiscoBox was evaluated on major benchmarks including COCO, PASCAL VOC 2012, and PF-PASCAL.

  • Instance Segmentation: On COCO, DiscoBox achieved 37.9% AP, setting a new state-of-the-art for weakly supervised methods and even outperforming some fully supervised baselines like Mask R-CNN.
  • Semantic Correspondence: On PF-PASCAL, it reached 59.3% [email protected], significantly exceeding previous weakly supervised approaches that lacked localization priors.
  • Efficiency: The complex teacher model is only utilized during training. Consequently, the student network maintains real-time inference speeds, making it practical for deployment.

Conclusion

DiscoBox demonstrates that bounding box supervision, when combined with structured self-ensembling, is sufficient to learn high-quality instance masks and semantic correspondences. By bridging the gap between these two tasks, the framework provides a scalable solution for dense visual understanding. Future work may extend this approach to fully unsupervised object discovery and 3D vision tasks where pixel-level correspondence is vital.

Summary updated

Abstract

We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentation and semantic correspondence are jointly guided by a structured teacher in addition to the bounding box supervision. The teacher is a structured energy model incorporating a pairwise potential and a cross-image potential to model the pairwise pixel relationships both within and across the boxes. Minimizing the teacher energy simultaneously yields refined object masks and dense correspondences between intra-class objects, which are taken as pseudo-labels to supervise the task network and provide positive/negative correspondence pairs for dense constrastive learning. We show a symbiotic relationship where the two tasks mutually benefit from each other. Our best model achieves 37.9% AP on COCO instance segmentation, surpassing prior weakly supervised methods and is competitive to supervised methods. We also obtain state of the art weakly supervised results on PASCAL VOC12 and PF-PASCAL with real-time inference.
BibTeX
@inproceedings{lan2021discobox,
  author = {Shiyi Lan and Zhiding Yu and Christopher Bongsoo Choy and Subhashree Radhakrishnan and Guilin Liu and Yuke Zhu and Larry S. Davis and Anima Anandkumar},
  title = {{DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=jiGVu31eHI0},
}
Figure