Abstract
This paper aims to reduce the time to annotate images for panoptic segmentation, which requires annotating segmentation masks and class labels for all object instances and stuff regions. We formulate our approach as a collaborative process between an annotator and an automated assistant who take turns to jointly annotate an image using a predefined pool of segments. Actions performed by the annotator serve as a strong contextual signal. The assistant intelligently reacts to this signal by annotating other parts of the image on its own, which reduces the amount of work required by the annotator. We perform thorough experiments on the COCO panoptic dataset, both in simulation and with human annotators. These demonstrate that our approach is significantly faster than previous machine-assisted interfaces and 2.4x to 5x faster than manual polygon drawing. Finally, we show on ADE20k that our method can be used to efficiently annotate new datasets, bootstrapping from a very small amount of annotated data.
Introduction
Panoptic segmentation is a demanding computer vision task that requires labeling every single pixel in an image with both a semantic class (e.g., "sky", "road") and an object instance ID (e.g., "car 1", "car 2"). Because it is so dense, creating ground-truth datasets is incredibly expensive. For example, it takes a human about 19 minutes to annotate a single image for the COCO dataset, and up to 1.5 hours for Cityscapes.
Recent tools have attempted to speed this up by having a neural network pre-generate a pool of segment proposals, which the human then selects and corrects. However, the human still performs most of the labor. This research introduces a Collaborative Assistant (CA) that doesn't just offer options but actively reacts to human choices to fill in the remaining segments automatically.
Methods: The Collaborative Loop
The authors frame annotation as a turn-based collaboration. The system builds on the Fluid Annotation interface, utilizing Mask R-CNN to generate approximately 1,000 overlapping segment proposals per image.
The core innovation is the CA's use of human actions as high-confidence contextual clues. When a human adds a segment or changes a label, that action is added to a Fixed Set. A context model—combining graph-convolutional and fully connected networks—evaluates the remaining unapproved proposals based on this Fixed Set. Because human actions provide unambiguous ground truth, the CA can make highly accurate inferences. For instance, correcting a segment from "sand" to "snow" allows the CA to automatically update nearby background segments to "snow" and identify objects like "snowboards."
Key Results
The system was evaluated on the COCO Panoptic and ADE20k datasets, demonstrating significant efficiency gains over manual methods.
- Efficiency Gains: In human user studies on COCO, the collaborative assistant was 5x faster than traditional manual polygon drawing (67 seconds vs. 338 seconds per image). While the initial quality was slightly lower (0.59 vs 0.74 PQ), adding a manual brush-tool refinement stage brought the quality to 0.73 while remaining 2.4x faster than manual drawing.
- Contextual Power: Simulations showed that the collaborative system reached a target panoptic quality score using 26% fewer human actions than non-collaborative baselines.
- Rapid Adaptation: The system demonstrated strong generalization. By fine-tuning on just 5% of the ADE20k dataset, the assistant annotated new images 12% to 19% faster than the baseline.
Conclusion and Limitations
The primary limitation is the reliance on pre-computed Mask R-CNN segments. If the initial model fails to generate a proposal with perfect boundaries, the assistant cannot correct the edges, leading to a quality cap unless manual refinement is used. However, the significant speedup demonstrates that human-AI collaboration is a viable path for scaling dense image annotation. Future work may integrate boundary refinement directly into the interactive loop.
Summary updated