SegBuilder: A Semi-Automatic Annotation Tool for Segmentation

Md Alimoor Reza, Eric Manley, Sean Chen, Sameer Chaudhary, Jacob ElafrosWACV 2025

TL;DR

This paper introduces SegBuilder, a novel semi-automatic tool that significantly speeds up the laborious process of semantic segmentation annotation by integrating the powerful Segment Anything Model (SAM). SegBuilder allows users to quickly label SAM-generated segments and offers intuitive tools for manual correction, making it particularly effective for challenging scenarios like camouflage and part-based segmentation.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task in computer vision with applications ranging from autonomous driving to medical imaging, requires vast amounts of pixel-level annotated data for training deep learning models. Traditionally, this annotation process is manual, involving drawing polygons around objects, which is incredibly time-consuming, labor-intensive, and requires significant user expertise. While recent advancements like the Segment Anything Model (SAM) can automatically generate object masks, these masks often lack class labels and can be imperfect, especially in complex visual scenarios. Existing annotation tools either don't integrate with models like SAM or require costly subscriptions, posing a barrier for researchers. This paper addresses the need for an efficient, accessible, and robust tool that bridges the gap between automated segmentation and accurate, human-verified annotations.

Methods

SegBuilder is a web-based, semi-automatic annotation tool that leverages the Segment Anything Model (SAM) as its core. The workflow involves:

  • SAM-generated Segments: SAM is used to automatically generate initial object masks for an image.
  • Human-in-the-Loop Labeling: Annotators are presented with these SAM-generated masks and can quickly assign labels from a predefined list using a simple drop-down menu.
  • Manual Correction and Refinement: For segments missed or inaccurately generated by SAM, SegBuilder provides a free-form drawing tool. This allows users to create custom polygons, which can then be converted into masks and labeled. This feature is crucial for handling challenging cases like shadows, camouflage, and part-based segmentation.
  • Project Management: The tool offers features for organizing images into projects, managing class labels with associated colors, and exporting annotations.

The system is built using Dash for the frontend and Flask for the backend, designed for easy deployment via Docker.

Datasets

  • UWSv2 ("Underwater Segmentation version 2"): A newly introduced dataset of 635 images with dense pixel-level annotations. This dataset expands upon existing underwater datasets by adding greater diversity in animal categories, including various fish species and new animals like beavers, dugongs, and sea snakes. It also includes 8 background categories.
  • PartImageNet [20]: Used for evaluating part-based segmentation capabilities.
  • Adaptive Camouflaged Dataset (ACD1K) [19] and SBU Shadow dataset [6]: Used to demonstrate SegBuilder's effectiveness in handling camouflage and shadow scenarios.

Key Insights and Ideas

  • SAM is a powerful starting point, but not a complete solution: While SAM excels at generating object masks, human oversight and refinement are essential for accurate, class-labeled segmentation, especially in complex scenes.
  • Semi-automatic annotation is key to efficiency: Combining automated segmentation with targeted human correction drastically reduces annotation time compared to purely manual methods.
  • Intuitive manual editing is crucial: The free-form drawing tool in SegBuilder is vital for addressing SAM's limitations, allowing annotators to easily correct errors and capture fine-grained details.
  • Accessibility matters for research: Providing an open-source, web-based tool like SegBuilder lowers the barrier for researchers to create high-quality datasets, especially when compared to subscription-based alternatives.

Key Results

  • Annotation Speedup: SegBuilder achieved an average annotation time of 1 minute and 9 seconds per image, which is 3.34x faster than the baseline tool Label Studio (3 minutes and 50 seconds).
  • Semantic Segmentation Performance: When used to train a HRNetV2 model on the new UWSv2 dataset, SegBuilder-annotated data achieved a mean IoU of 62.54%.
  • Challenging Scenarios: SegBuilder demonstrated significantly improved accuracy in segmenting camouflaged objects and scenes with shadows compared to SAM alone, as shown qualitatively in Figures 7 and 8.
  • Part Segmentation: SegBuilder consistently produced more accurate part segmentations than SAM alone on the PartImageNet dataset, capturing smaller or less distinct parts effectively.

Limitations and Future Work

  • SAM's inherent limitations: SegBuilder relies on SAM's output, so its effectiveness is still bound by SAM's underlying capabilities and potential biases.
  • Computational resources for SAM: Generating SAM masks can be computationally intensive, and the paper mentions a separate utility for pre-generating these masks.
  • Future Work: The paper suggests that SegBuilder can be used to train semantic segmentation models, and the tool's public release is intended to encourage collaborative improvements and future development. The authors also highlight the potential for SegBuilder to be applied to a wide range of applications and image data types.

Summary updated

BibTeX
@inproceedings{reza2025segbuilder,
  author = {Md Alimoor Reza and Eric Manley and Sean Chen and Sameer Chaudhary and Jacob Elafros},
  title = {{SegBuilder: A Semi-Automatic Annotation Tool for Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure