Quality control in crowdsourced object segmentation

Ferran Cabezas, Axel Carlier, Vincent Charvillat, Amaia Salvador, Xavier Giró-i-Nieto

TL;DR

This paper tackles the "noise" problem in crowdsourced object segmentation by combining computer vision algorithms with human behavior analysis. By implementing a novel superpixel-based weighting system that accounts for individual worker expertise, the authors improved segmentation accuracy (Jaccard Index) from a baseline of 0.14 to a robust 0.86, nearly matching expert-level performance.

What Problem Does This Paper Address?

Object segmentation—the task of defining exactly which pixels belong to an object—is a cornerstone of computer vision. While expert researchers can produce high-quality segments, hiring them is expensive and slow. Crowdsourcing platforms like Amazon Mechanical Turk or Microworkers offer a cheaper, faster alternative, but there’s a catch: "workers" are often non-experts who may provide sloppy, hurried, or even intentionally "scammy" data to maximize their earnings.

The authors address the challenge of quality control: How do we take messy, contradictory "clicks" from a crowd of unvetted users and turn them into a precise object mask?

Methods

The researchers explored three distinct strategies for cleaning up crowdsourced data:

  1. Filtering Clicks (Spatial Consistency): They used superpixel algorithms (SLIC and Felzenszwalb) to group pixels into visually similar regions. If a single superpixel contained both "foreground" and "background" clicks from users, it was flagged as a conflict. They tested two fixes: keeping only the majority label or discarding the conflicting clicks entirely.
  2. Filtering Users (Performance-Based): Users were asked to segment a small "gold standard" set of images with known ground truths. The authors then ranked users by their error rates and Jaccard indices, experimenting with discarding the bottom-performing workers.
  3. Data Weighting (The "Soft" Approach): Instead of binary filtering (keeping or throwing away data), they developed Algorithm 1. This approach assigns a quality score to each user based on their performance on the gold standard images. These scores are then used to weight every click. If a high-quality user clicks an area, it carries more weight in the final segmentation than a click from a low-quality user. This allows the system to benefit from the "wisdom of the crowd" without being derailed by a few bad actors.

Datasets

  • Test Set: 96 images (100 tasks) from the DCU dataset, which is a subset of the Berkeley Segmentation Database.
  • Training/Gold Standard Set: 5 images from the PASCAL VOC dataset, used to establish user quality scores.
  • Crowdsourcing Platform: Data was collected from 20 workers on Microworkers.com, who performed 105 tasks each.

Key Insights and Ideas

  • Context is King: A click in isolation is just a coordinate. However, when mapped to a superpixel, it gains semantic meaning. If a user clicks "foreground" in a region the computer sees as a distinct visual unit, that click is likely reliable.
  • The "Spammer" vs. the "Amateur": The authors found that a user's average Jaccard Index (how well they define the whole object) is a much better predictor of quality than their raw "click error rate." Some users might click slightly outside a boundary (high error) but still provide a very helpful overall shape.
  • Don't Throw the Baby Out with the Bathwater: Hard filtering (discarding users) can be too aggressive, especially when you have few labels to begin with. The "soft" weighting approach proved superior because it retains the maximum amount of information while naturally suppressing noise.

Key Results

  • The Baseline: Raw, unfiltered crowdsourced traces yielded a dismal Jaccard Index of 0.14.
  • The Expert Benchmark: Expert researchers using the same tools achieved a Jaccard Index of 0.93.
  • Click Filtering: Simply using superpixels to resolve conflicts boosted the score to 0.24 (a 71% improvement over baseline, but still objectively poor).
  • User Filtering: By keeping only the top 2 users, the score jumped to 0.90. However, relying on only two users is not statistically robust or scalable.
  • The Winning Solution: The proposed superpixel-based weighting algorithm achieved a Jaccard Index of 0.86 using the entire pool of workers. This represents a massive leap from the baseline and approaches expert-level quality without needing to manually vet every worker.

Limitations and Future Work

The primary limitation is the reliance on a "gold standard" dataset. To calculate user quality scores, you must already have perfect ground truth for a subset of your images. In a purely "cold start" scenario where no ground truth exists, this method would require a different way to estimate user expertise (perhaps through inter-user consistency).

The authors suggest that future work could involve integrating these quality control measures into the interactive tool itself, providing real-time feedback to workers to "train" them into becoming higher-quality contributors as they work.

Summary updated

Abstract

This paper explores processing techniques to deal with noisy data in crowdsourced object segmentation tasks. We use the data collected with Click'n'Cut, an online interactive segmentation tool, and we perform several experiments towards improving the segmentation results. First, we introduce different superpixel-based techniques to filter users' traces, and assess their impact on the segmentation result. Second, we present different criteria to detect and discard the traces from potential bad users, resulting in a remarkable increase in performance. Finally, we show a novel superpixel-based segmentation algorithm which does not require any prior filtering and is based on weighting each user's contribution according to his/her level of expertise.
BibTeX
@inproceedings{cabezas2015quality,
  author = {Ferran Cabezas and Axel Carlier and Vincent Charvillat and Amaia Salvador and Xavier Giró-i-Nieto},
  title = {{Quality control in crowdsourced object segmentation}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=9dahex1hhx},
}

Loading related content...

Figure