Large-Scale Interactive Object Segmentation With Human Annotators

Rodrigo Benenson, Stefan Popov, Vittorio Ferrari

Abstract

Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We released this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3x faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.

Introduction: The Annotation Bottleneck

In computer vision, high-quality data is essential, but producing pixel-level instance segmentation masks is notoriously expensive. While bounding boxes are relatively cheap, traditional manual mask creation—often involving clicking dozens of points to define a polygon—can take over a minute per object. This "annotation bottleneck" hinders the creation of the massive datasets required for modern deep learning models.

To address this, researchers have developed a high-efficiency interactive system that pairs human annotators with deep learning models. Instead of a human performing all the work, a model proposes a mask, and a human annotator simply provides "corrective clicks" to fix errors.

Methods: The Interactive Pipeline

The system utilizes a multi-stage interactive pipeline designed for efficiency and scale:

  1. Initial Mask Generation ($M_b$): The process begins with a bounding box and the image. A model generates an initial "best guess" mask.
  2. Corrective Clicks ($M_{b+c}$): An annotator reviews the mask and places corrective clicks: green for foreground areas that should be included, and red for background areas that should be removed.
  3. Iterative Refinement: A second model incorporates these clicks to update the mask. This process repeats over several rounds. The study found that a "4x3" setup—up to 4 clicks across 3 rounds—provides an optimal balance of speed and quality.
The iterative interactive segmentation pipeline: starting from a bounding box, a human provides corrective clicks (green for foreground, red for background) to refine the mask over several rounds.
  1. Quality Ranking ($M_r$): A novel Ranking Model (a decision forest) predicts the quality (Intersection over Union, or IoU) of the final mask without ground truth. It uses indirect signals such as the number of clicks, the distance of clicks from the mask boundary, and the magnitude of mask changes between rounds.

Datasets and Scalability

This method was used to create the largest instance segmentation dataset to date:

  • OpenImages V4: The authors produced 2.5 million instance masks across 300 categories.
  • COCO: Portions of COCO were re-annotated to benchmark the method against traditional polygon-based tools.
  • ADE20k: Used for class-agnostic pre-training of the segmentation models.
The system is class-agnostic, capable of segmenting diverse objects like this lime by leveraging pre-training on broad datasets.

Key Insights

Exploration of the interactive design space yielded several important findings:

  • Binary Disks are Effective: Encoding human clicks as simple binary disks outperformed more complex representations like Gaussian heatmaps.
  • Center-Clicking Strategy: Clicking in the center of an error region provides a more robust signal to the model than clicking on the boundary.
  • Class-Agnosticism: A single model trained on diverse objects (e.g., from ADE20k) performs as well as specialized models, facilitating massive scalability across hundreds of categories.
  • Iteration over Volume: It is more effective to provide a few clicks, allow the model to update, and then provide more clicks, rather than providing all clicks at once.

Results: Speed and Quality

The interactive approach demonstrated significant advantages over traditional methods:

  • 3x Speedup: The method is three times faster than the polygon drawing tools used for the original COCO dataset.
  • Higher Quality: Despite the speed, interactive masks achieved 84% mIoU, outperforming COCO’s original polygon annotations (82% mIoU) when compared against a high-precision "gold standard."
  • Automated Quality Control: The Ranking Model ($M_r$) effectively identifies high-quality subsets; for instance, it can automatically select a data subset where 70% of masks have an IoU of 90% or higher.

Limitations and Future Work

Challenges remain for complex structures, such as very thin or fragmented objects (e.g., fine tree branches), which are difficult for current models to capture perfectly. Additionally, human annotators sometimes fix tiny details instead of moving on, slightly impacting efficiency. Future research could explore active learning, where models "ask" for clicks in specific areas, or improved handling of occlusions and transparency.

Summary updated

Abstract

Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We released this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3x faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.
BibTeX
@inproceedings{benenson2019large,
  author = {Rodrigo Benenson and Stefan Popov and Vittorio Ferrari},
  title = {{Large-Scale Interactive Object Segmentation With Human Annotators}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=xc9B2X5_7i},
}
Figure