Abstract
Manually annotating object segmentation masks is a notoriously labor-intensive task. Interactive object segmentation methods offer a more efficient alternative by enabling a collaborative loop between human annotators and machine learning models. This paper explores the design space of deep interactive segmentation models, executes a large-scale annotation campaign producing 2.5 million instance masks for the OpenImages dataset, and introduces a technique for automatically estimating mask quality using indirect process signals. Our results show that this interactive approach is three times faster than traditional polygon drawing while providing superior mask quality.
The Annotation Bottleneck
Creating pixel-perfect segmentation masks is often considered the most difficult stage of data annotation. While bounding boxes are relatively inexpensive to produce, masks require significant manual effort. Traditional tools typically rely on "polygon drawing," where a human must click dozens of points to trace an object's boundary. This method is not only slow but often results in jagged, imprecise edges that fail to capture the true geometry of the object.
To address this, we propose a shift from manual drawing to an interactive collaboration. By asking how humans can provide only the most necessary corrections, we allow a model to predict the mask while the human provides "corrective clicks" to resolve errors.
Methods: The Interactive Pipeline
Our proposed system utilizes a multi-stage pipeline designed for maximum efficiency:
- Initial Segmentation ($M_b$): A convolutional neural network (CNN) generates an initial mask based on a user-provided bounding box.
- Interactive Refinement ($M_{b+c}$): An annotator reviews the initial mask and places up to four corrective clicks per round on incorrectly labeled areas. A second model incorporates these clicks to refine the mask. This process typically repeats for three rounds to achieve high precision.
- Quality Ranking ($M_r$): To manage variability in human input, we trained a "Ranking Model" (a decision forest) to estimate the Intersection-over-Union (IoU) of the final mask. This model uses indirect signals—such as the number of clicks, time taken, and mask delta between rounds—to predict quality without needing to process the image itself.
Key Insights and Design Choices
Through systematic simulation and real-world testing, we identified several critical insights for building effective interactive tools:
- Region Clicks over Boundary Clicks: We found that clicking the center of an error region (e.g., a hole that should be filled) is more robust to human noise and provides a clearer signal to the model than clicking precisely on the boundary.
- Simplicity in Encoding: When feeding human clicks into the neural network, simple binary disks (small circles around the click) outperformed more complex distance transforms or Gaussian distributions.
- The Power of Interaction: It is significantly more efficient to perform multiple rounds of few clicks (where the model updates between rounds) than to provide many clicks in a single batch. The model's ability to extrapolate from sequential human input reduces the total workload.
- Class-Agnostic Performance: Surprisingly, a model trained on a diverse set of objects performs as well as a class-specialized model after just a few rounds of interaction, making it a versatile tool for any object category.
Results and Impact
Our large-scale deployment on the OpenImages V4 dataset resulted in 2.5 million new instance masks across 300 categories, creating the largest public dataset for instance segmentation to date.
Benchmark comparisons on the COCO dataset demonstrate that our interactive system is 3x faster than traditional polygon tools. Despite the increased speed, the resulting masks achieved 84% mIoU, surpassing the 82% mIoU of COCO's manually drawn polygons. Furthermore, our ranking model ($M_r$) successfully identifies high-quality data; for instance, it can automatically filter the top 70% of masks to achieve an average of 90% IoU.
Limitations and Future Work
While highly effective, some challenges remain. Complex, thin, or translucent structures (like jellyfish tentacles) are still difficult for current models to segment perfectly. Additionally, "annotation policy" remains a bottleneck; deciding whether attached items (like a backpack on a person) should be included in a mask requires extensive human training. Future research will focus on using the indirect signals from our ranking model to decide in real-time when an object has received enough clicks, further optimizing the human-in-the-loop process.
Summary updated