Abstract
Models based on deep convolutional neural networks (CNN) have significantly improved the performance of semantic segmentation. However, learning these models requires a large amount of training images with pixel-level labels, which are very costly and time-consuming to collect. In this paper, we propose a method for learning CNN-based semantic segmentation models from images with several types of annotations that are available for various computer vision tasks, including image-level labels for classification, box-level labels for object detection and pixel-level labels for semantic segmentation. The proposed method is flexible and can be used together with any existing CNN-based semantic segmentation networks. Experimental evaluation on the challenging PASCAL VOC 2012 and SIFTflow benchmarks demonstrate that the proposed method can effectively make use of diverse training data to improve the performance of the learned models.
The Data Bottleneck in Semantic Segmentation
Semantic segmentation—the task of assigning a category label to every single pixel in an image—is a cornerstone of computer vision. However, its progress is bottlenecked by data. Training a deep convolutional neural network (CNN) for this task typically requires thousands of images with pixel-perfect "ground truth" labels.
Collecting these labels is incredibly slow and expensive; for context, drawing a bounding box is roughly 15 times faster than tracing a pixel-level mask. Meanwhile, massive datasets already exist for image classification (tags) and object detection (boxes). The challenge this paper tackles is: How can we build a segmentation model that learns from all these different types of data simultaneously?
A Unified Framework for Diverse Supervision
The authors propose a general framework that can be wrapped around any existing segmentation architecture (they test it with FCN and DeepLab). The core innovation is the Annotation-Specific Loss Module, which branches into three paths depending on what label is available for a given training image:
- Image-Level Branch: For images that only have a tag (e.g., "dog"), the model uses Global Average Pooling (GAP) to collapse the spatial feature maps into a single vector. It then applies a multi-label binary cross-entropy loss to teach the network to recognize the presence of objects without knowing their location.
- Box-Level Branch: For images with bounding boxes, the authors use Ultrametric Contour Maps (UCM)—a classical computer vision technique—to estimate the object's shape within the box. They handle overlapping boxes by using a "Soft Segmentation" loss, which weights pixels based on how many candidate object masks they belong to, effectively managing spatial ambiguity.
- Pixel-Level Branch: For images with full masks, the model uses standard supervised learning (softmax loss).
During training, the model processes mini-batches where each image might have a different type of label, activating only the relevant branch and backpropagating the signal to the shared "backbone" network.
Experimental Results
The researchers evaluated their approach on two major benchmarks: PASCAL VOC 2012 (20 object classes) and SIFT-flow (33 semantic categories). To test the model's robustness, they created various training splits with different ratios of pixel-level, box-level, and image-level data.
Key results include:
- Performance Boost: On PASCAL VOC 2012, adding weak supervision (boxes and tags) consistently improved results. For example, using a FCN backbone with a 1:5:10 data split, the Mean Intersection over Union (mIU) jumped from 51.08% (pixel-only) to 57.51% (diverse supervision).
- Closing the Gap: In the 1:1:1 data setting, the proposed model achieved performance within 2% of the "upper bound"—a theoretical model trained with expensive pixel-level labels for every image in the set.
- Efficiency: The model significantly outperformed previous state-of-the-art weakly supervised methods (like BoxSup and WSSL) when compared under similar conditions, particularly in how effectively it utilized the box-level information.
Insights and Future Directions
The study reveals a Synergy of Weakness: different labels provide complementary information. Image-level tags help the model identify what categories are present, while box-level labels provide crucial "priors" on where those objects are and their approximate scale. Furthermore, the method is Architecture Agnostic, meaning it can be applied to newer backbones as they emerge.
However, the approach has limitations. The box-to-mask conversion relies on UCM, a non-learned algorithm; if UCM fails, the "pseudo-labels" suffer. Additionally, while the model reduces the need for pixel data, it still requires some high-quality masks to "anchor" the learning process. Future work could extend this framework to point-level supervision or scribbles, further lowering the barrier to entry for high-performance semantic segmentation.
Summary updated