Task Configuration Impacts Annotation Quality and Model Training Performance in Crowdsourced Image Segmentation

Benjamin R Bauchwitz, Mary CummingsWACV 2025

TL;DR

This paper introduces a new tool (CSET) and dataset (VACES) to systematically study how different crowdsourcing task configurations affect image segmentation annotation quality. They found that paying annotators per object, using a paintbrush tool, and requiring some task complexity significantly improve annotation quality and, consequently, model training performance.

What Problem Does This Paper Address?

Industrial applications like autonomous driving and medical imaging heavily rely on accurate image segmentation. This typically requires large, meticulously annotated datasets. However, there's a lack of standardization in how these annotations are produced, especially when using crowdsourcing platforms like Amazon Mechanical Turk. Developers often opt for convenience, which can lead to unintended consequences for annotation quality and, ultimately, the performance of the trained models. This paper aims to bridge this gap by rigorously investigating how various aspects of the annotation task configuration—such as payment schemes, drawing tools, time limits, and class diversity—impact the quality of the resulting segmentation masks and their effectiveness in training deep learning models.

Methods

The authors developed the Crowdsource Segmentation Experiment Tool (CSET), a web application designed for conducting crowdsourced image segmentation experiments. CSET is built on the "Turkey" MTurk plugin and offers a flexible user interface with various drawing tools (points, lines, bounding boxes, polygon, outline, and paintbrush). It also includes editing capabilities and robust logging of user interactions.

To create their dataset, they used CSET to run experiments on Amazon Mechanical Turk, varying several task configurations:

  • Compensation: Paid per image vs. paid per object.
  • Time Limit: No limit vs. a 3-minute limit per image.
  • Drawing Tools: Full suite vs. restricted to a single tool (polygon, outline, or paintbrush).
  • Class Diversity: Images with single classes vs. images with multiple classes.

They then evaluated annotation quality using the Intersection over Union (IoU) metric against ground truth from existing benchmark datasets (PASCAL VOC, COCO, Berkeley Deep Drive). Finally, they trained several semantic segmentation models on the generated annotations to assess the impact on model performance.

Datasets

  • VACES (Variable Annotations from a Crowdsourcing Experiment on Segmentation): A newly created dataset consisting of 17,600 images from PASCAL VOC 2012, COCO, and Berkeley Deep Drive (BDD) datasets, annotated under 8 different crowdsourcing task configurations. This dataset is designed to benchmark segmentation model robustness across various annotation patterns.
  • Benchmark Datasets (for ground truth): PASCAL VOC 2012, COCO, and Berkeley Deep Drive (BDD).

Key Insights and Ideas

  • Pay Per Object is Key: Paying annotators per annotated object, even if the total payout is the same as per image, significantly boosts annotation quality. This suggests that annotators are incentivized by the potential for higher earnings on more complex objects, leading to more careful work.
  • Paintbrush Power: The paintbrush tool, despite being more time-consuming, consistently yielded the highest quality annotations. This highlights the trade-off between speed and precision in annotation tasks.
  • Complexity Prevents Complacency: Counterintuitively, tasks with higher class diversity (more complex images) led to better annotation quality. This suggests that overly simple tasks can lead to boredom and reduced engagement, negatively impacting performance.
  • Model Robustness: While annotation errors do impact model training, the tested models showed a surprising degree of resilience. Even annotations with lower IoU scores could still lead to models achieving a significant portion of the performance seen when trained on ground truth.

Key Results

  • Annotation Quality:
    • Paying per object improved IoU scores compared to paying per image (e.g., 0.58 vs. 0.56 for configurations B vs. A).
  • The paintbrush tool (G) achieved the highest mean IoU (0.60), while the outline tool (F) resulted in the lowest (0.54).
  • Increased class diversity (D2) led to higher IoU than single-class images (D1) (0.58 vs. 0.54).
  • Annotation completion times were generally correlated with IoU scores; slower times often meant higher quality.
  • Model Training Performance:
    • Models trained on the best crowdsourced annotations (paintbrush tool, G) achieved performance 4-6 percentage points higher than those trained on the worst (outline tool, F).
  • Even the best crowdsourced annotations resulted in a 12-22% reduction in model performance compared to training on ground truth.
  • Models trained on paintbrush annotations (average IoU of 0.60) achieved roughly 80% of the performance of models trained on ground truth.

Limitations and Future Work

  • Hybrid Systems: The study focused on purely manual annotation and did not assess the impact of task configurations on hybrid systems that combine human and automated annotation. Future work should explore how these configurations interact with automated assistance.
  • Spatial Properties of Errors: While the study identified general trends, a deeper investigation into the specific spatial properties of annotation errors and their impact on training is needed.
  • Generalizability: The findings are based on specific task configurations and models. Further research could explore a wider range of configurations, models, and application domains.
  • Annotator Motivation: While the study touched on engagement, a more in-depth understanding of how different factors influence annotator motivation and sustained performance would be valuable.

Summary updated

BibTeX
@inproceedings{bauchwitz2025task,
  author = {Benjamin R Bauchwitz and Mary Cummings},
  title = {{Task Configuration Impacts Annotation Quality and Model Training Performance in Crowdsourced Image Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure