Large-scale interactive object segmentation with human annotators

Rodrigo Benenson, Stefan Popov, Vittorio Ferrari

Abstract

Manually annotating object segmentation masks is a notoriously labor-intensive task. Interactive object segmentation methods offer a more efficient alternative by enabling a collaborative loop between human annotators and machine learning models. This paper explores the design space of deep interactive segmentation models, executes a large-scale annotation campaign producing 2.5 million instance masks for the OpenImages dataset, and introduces a technique for automatically estimating mask quality using indirect process signals. Our results show that this interactive approach is three times faster than traditional polygon drawing while providing superior mask quality.

The Annotation Bottleneck

Creating pixel-perfect segmentation masks is often considered the most difficult stage of data annotation. While bounding boxes are relatively inexpensive to produce, masks require significant manual effort. Traditional tools typically rely on "polygon drawing," where a human must click dozens of points to trace an object's boundary. This method is not only slow but often results in jagged, imprecise edges that fail to capture the true geometry of the object.

To address this, we propose a shift from manual drawing to an interactive collaboration. By asking how humans can provide only the most necessary corrections, we allow a model to predict the mask while the human provides "corrective clicks" to resolve errors.

Methods: The Interactive Pipeline

Our proposed system utilizes a multi-stage pipeline designed for maximum efficiency:

  1. Initial Segmentation ($M_b$): A convolutional neural network (CNN) generates an initial mask based on a user-provided bounding box.
  2. Interactive Refinement ($M_{b+c}$): An annotator reviews the initial mask and places up to four corrective clicks per round on incorrectly labeled areas. A second model incorporates these clicks to refine the mask. This process typically repeats for three rounds to achieve high precision.
  3. Quality Ranking ($M_r$): To manage variability in human input, we trained a "Ranking Model" (a decision forest) to estimate the Intersection-over-Union (IoU) of the final mask. This model uses indirect signals—such as the number of clicks, time taken, and mask delta between rounds—to predict quality without needing to process the image itself.
Iterative refinement process: starting from a bounding box, the annotator provides corrective clicks to guide the model toward a precise mask across multiple rounds.

Key Insights and Design Choices

Through systematic simulation and real-world testing, we identified several critical insights for building effective interactive tools:

  • Region Clicks over Boundary Clicks: We found that clicking the center of an error region (e.g., a hole that should be filled) is more robust to human noise and provides a clearer signal to the model than clicking precisely on the boundary.
  • Simplicity in Encoding: When feeding human clicks into the neural network, simple binary disks (small circles around the click) outperformed more complex distance transforms or Gaussian distributions.
  • The Power of Interaction: It is significantly more efficient to perform multiple rounds of few clicks (where the model updates between rounds) than to provide many clicks in a single batch. The model's ability to extrapolate from sequential human input reduces the total workload.
  • Class-Agnostic Performance: Surprisingly, a model trained on a diverse set of objects performs as well as a class-specialized model after just a few rounds of interaction, making it a versatile tool for any object category.
Interactive segmentation inputs: the model uses bounding boxes and positive/negative clicks (green and red dots) to generate and refine object masks.

Results and Impact

Our large-scale deployment on the OpenImages V4 dataset resulted in 2.5 million new instance masks across 300 categories, creating the largest public dataset for instance segmentation to date.

Benchmark comparisons on the COCO dataset demonstrate that our interactive system is 3x faster than traditional polygon tools. Despite the increased speed, the resulting masks achieved 84% mIoU, surpassing the 82% mIoU of COCO's manually drawn polygons. Furthermore, our ranking model ($M_r$) successfully identifies high-quality data; for instance, it can automatically filter the top 70% of masks to achieve an average of 90% IoU.

Limitations and Future Work

While highly effective, some challenges remain. Complex, thin, or translucent structures (like jellyfish tentacles) are still difficult for current models to segment perfectly. Additionally, "annotation policy" remains a bottleneck; deciding whether attached items (like a backpack on a person) should be included in a mask requires extensive human training. Future research will focus on using the indirect signals from our ranking model to decide in real-time when an object has received enough clicks, further optimizing the human-in-the-loop process.

Summary updated

Abstract

Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We plan to release this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3 times faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.
BibTeX
@inproceedings{benenson2019large,
  author = {Rodrigo Benenson and Stefan Popov and Vittorio Ferrari},
  title = {{Large-scale interactive object segmentation with human annotators}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=xc9B2X5_7i},
}
Figure