A Crowdsourcing Repeated Annotations System for Visual Object Detection

Yucheng Hu, Zhonghong Ou, Xiangyu Xu, Meina Song

Abstract

As a fundamental task in compute vision, object detection has been developed rapidly driven by the deep learning. The lack of a large number of images with ground truth annotations has become a chief obstacle to object detection applications in many fields. Eliciting labels from crowds is a potential way to obtain large labeled data. Nonetheless, existing crowdsourced techniques, e.g., Amazon Mechanical Turk (MTurk), often fail to guarantee the quality of the annotations, which have a bad influence on the accuracy of the deep detector. A variety of methods have been developed for ground truth inference and learning from crowds. In this paper, we study strategies to crowd-source repeated labels in support for these methods. The core challenge of building such a system is to reduce the difficulty to annotate multiple objects of interest and improve the data quality as much as possible. We present a system that adopts the turn-based annotation mechanism and consists of three simple sub-tasks: a single object annotation, a quality verification task and a coverage verification task. Experimental results demonstrate that our system is scalable, accurate and can assist the detector of obtaining higher accuracy.
BibTeX
@inproceedings{hu2019crowdsourcing,
  author = {Yucheng Hu and Zhonghong Ou and Xiangyu Xu and Meina Song},
  title = {{A Crowdsourcing Repeated Annotations System for Visual Object Detection}},
  booktitle = {ICVISP},
  year = {2019},
  doi = {10.1145/3387168.3387242},
}

Loading related content...

Figure