Quality Control in Crowdsourced Object Segmentation

Ferran Cabezas, Axel Carlier, Amaia Salvador, Xavier Giró-i-Nieto, Vincent Charvillat

Abstract

Crowdsourcing offers a scalable and cost-effective solution for object segmentation tasks in computer vision. However, the inherent noise in data provided by non-expert workers often compromises the quality of the final segmentation masks. This paper investigates various processing techniques to mitigate this noise, leveraging the "Click'n'Cut" interactive tool. We evaluate superpixel-based spatial filtering, performance-based user filtering, and a novel soft-weighting algorithm. Our results demonstrate that by accounting for individual worker expertise and spatial context, crowdsourced segmentation accuracy can be improved from a baseline Jaccard Index of 0.14 to 0.86, approaching expert-level performance (0.93).

Introduction

Object segmentation—the process of delineating the precise boundaries of objects within an image—is fundamental to many computer vision applications. While expert-annotated datasets are highly accurate, they are expensive and time-consuming to produce. Crowdsourcing platforms like Amazon Mechanical Turk provide a viable alternative, but the quality of contributions varies significantly. Workers may be inexperienced, hurried, or intentionally provide low-quality data. Effective quality control mechanisms are therefore essential to transform noisy, crowdsourced "clicks" into reliable object masks.

Methodology

We explored three primary strategies for enhancing the quality of crowdsourced segmentation data:

1. Spatial Consistency via Click Filtering

Individual clicks lack semantic context. To address this, we utilized superpixel algorithms (SLIC and Felzenszwalb) to group pixels into visually coherent regions. By mapping user clicks to these superpixels, we can identify conflicts—for instance, when a single superpixel contains both foreground and background labels. We tested methods to resolve these conflicts by either adopting the majority label or discarding the conflicting clicks entirely.

Superpixel-based context for individual clicks helps identify spatial inconsistencies.

2. Performance-Based User Filtering

Not all workers contribute equally. We established a "gold standard" set of images with known ground truths to evaluate user performance. Users were ranked based on their error rates and Jaccard indices. This allowed us to experiment with discarding data from the lowest-performing workers to improve the aggregate result.

3. Soft Data Weighting

Rather than binary filtering, we proposed a "soft" approach. Our algorithm assigns a quality score to each user based on their performance on the gold standard set. These scores serve as weights for their clicks in the final segmentation process. This method preserves the "wisdom of the crowd" while naturally attenuating the influence of unreliable contributors.

Experimental Setup

The study utilized a test set of 96 images from the DCU dataset (a subset of the Berkeley Segmentation Database) and a training set of 5 images from the PASCAL VOC dataset for user scoring. Data was collected from 20 workers on Microworkers.com, each performing 105 tasks.

Results and Discussion

The baseline Jaccard Index for raw, unfiltered crowdsourced data was a mere 0.14. In contrast, expert researchers achieved a benchmark of 0.93.

Click filtering using superpixels improved the score to 0.24. While this represents a 71% improvement over the baseline, it remains insufficient for high-quality applications. User filtering proved more effective; by retaining only the top two performers, the Jaccard Index reached 0.90. However, this approach is fragile and does not scale well, as it relies on a very small subset of the workforce.

Jaccard Index performance when filtering users by descending Jaccard index (blue) and ascending error rate (green).

The proposed soft-weighting algorithm achieved a robust Jaccard Index of 0.86 using the entire worker pool. This approach effectively bridges the gap between raw crowdsourced data and expert-level quality without the need for aggressive data discarding.

Conclusion

This research highlights the importance of combining computer vision techniques with human behavior analysis for quality control in crowdsourcing. By leveraging superpixels for spatial context and weighting contributions by estimated expertise, we can significantly enhance the reliability of crowdsourced object segmentation. Future work will focus on estimating user quality without prior ground truth and integrating real-time feedback into segmentation tools to improve worker performance during the task.

Summary updated

Abstract

This paper explores processing techniques to deal with noisy data in crowdsourced object segmentation tasks. We use the data collected with "Click'n'Cut", an online interactive segmentation tool, and we perform several experiments towards improving the segmentation results. First, we introduce different superpixel-based techniques to filter users' traces, and assess their impact on the segmentation result. Second, we present different criteria to detect and discard the traces from potential bad users, resulting in a remarkable increase in performance. Finally, we show a novel superpixel-based segmentation algorithm which does not require any prior filtering and is based on weighting each user's contribution according to his/her level of expertise.
BibTeX
@inproceedings{cabezas2015quality,
  author = {Ferran Cabezas and Axel Carlier and Amaia Salvador and Xavier Giró-i-Nieto and Vincent Charvillat},
  title = {{Quality Control in Crowdsourced Object Segmentation}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=9dahex1hhx},
}
Figure