Abstract
Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We released this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3x faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.
Introduction: The Annotation Bottleneck
In computer vision, high-quality data is essential, but producing pixel-level instance segmentation masks is notoriously expensive. While bounding boxes are relatively cheap, traditional manual mask creation—often involving clicking dozens of points to define a polygon—can take over a minute per object. This "annotation bottleneck" hinders the creation of the massive datasets required for modern deep learning models.
To address this, researchers have developed a high-efficiency interactive system that pairs human annotators with deep learning models. Instead of a human performing all the work, a model proposes a mask, and a human annotator simply provides "corrective clicks" to fix errors.
Methods: The Interactive Pipeline
The system utilizes a multi-stage interactive pipeline designed for efficiency and scale:
- Initial Mask Generation ($M_b$): The process begins with a bounding box and the image. A model generates an initial "best guess" mask.
- Corrective Clicks ($M_{b+c}$): An annotator reviews the mask and places corrective clicks: green for foreground areas that should be included, and red for background areas that should be removed.
- Iterative Refinement: A second model incorporates these clicks to update the mask. This process repeats over several rounds. The study found that a "4x3" setup—up to 4 clicks across 3 rounds—provides an optimal balance of speed and quality.
- Quality Ranking ($M_r$): A novel Ranking Model (a decision forest) predicts the quality (Intersection over Union, or IoU) of the final mask without ground truth. It uses indirect signals such as the number of clicks, the distance of clicks from the mask boundary, and the magnitude of mask changes between rounds.
Datasets and Scalability
This method was used to create the largest instance segmentation dataset to date:
- OpenImages V4: The authors produced 2.5 million instance masks across 300 categories.
- COCO: Portions of COCO were re-annotated to benchmark the method against traditional polygon-based tools.
- ADE20k: Used for class-agnostic pre-training of the segmentation models.
Key Insights
Exploration of the interactive design space yielded several important findings:
- Binary Disks are Effective: Encoding human clicks as simple binary disks outperformed more complex representations like Gaussian heatmaps.
- Center-Clicking Strategy: Clicking in the center of an error region provides a more robust signal to the model than clicking on the boundary.
- Class-Agnosticism: A single model trained on diverse objects (e.g., from ADE20k) performs as well as specialized models, facilitating massive scalability across hundreds of categories.
- Iteration over Volume: It is more effective to provide a few clicks, allow the model to update, and then provide more clicks, rather than providing all clicks at once.
Results: Speed and Quality
The interactive approach demonstrated significant advantages over traditional methods:
- 3x Speedup: The method is three times faster than the polygon drawing tools used for the original COCO dataset.
- Higher Quality: Despite the speed, interactive masks achieved 84% mIoU, outperforming COCO’s original polygon annotations (82% mIoU) when compared against a high-precision "gold standard."
- Automated Quality Control: The Ranking Model ($M_r$) effectively identifies high-quality subsets; for instance, it can automatically select a data subset where 70% of masks have an IoU of 90% or higher.
Limitations and Future Work
Challenges remain for complex structures, such as very thin or fragmented objects (e.g., fine tree branches), which are difficult for current models to capture perfectly. Additionally, human annotators sometimes fix tiny details instead of moving on, slightly impacting efficiency. Future research could explore active learning, where models "ask" for clicks in specific areas, or improved handling of occlusions and transparency.
Summary updated