WhichDog: A crowdsourced dataset including candidate set-based labelling

Iker Beñaran-Muñoz, Jerónimo Hernández-González, Aritz PérezZenodo 2022 · DOI · OpenAlex

Abstract

A dataset with crowdsourced labels for aggregation and supervised classification.<br> It contains 400 images of dogs from the Stanford Dogs dataset (http://vision.stanford.edu/aditya86/ImageNetDogs/). Images of dogs that belong to 32 different breeds (classes) are included. Annotators were asked to provide two types of labelling: full labelling (each labeler is allowed to provide a single label for each image) and candidate labelling (each labeler is allowed to provide a set of candidate labels for each image). It includes a total of 61227 annotations (30628 full and 30599 candidate) obtained from a set of 1028 different labelers. The labels were collected through the online crowdsourcing platform Amazon mTurk thanks to funds provided by the Basque Government through the Elkartek program (KK-2018/00071). The assignments were designed as sequences of 64 images that were given to the annotators. Each image in the sequence was provided together with a specific subset of possible labels (with the number of options ranging from 4 to 32), and a instruction for the annotator to perform a specific type of labelling (full or candidate). Each labeler performed at least one assignment. Not all the labelers completed the 64 annotations in their assignments.<br> <br> The file 'whichdog.zip' contains a folder ('images') with the 400 images of dogs, a text file ('breed_names.txt') that indicates the names of the different breeds and their assigned label (a number in the interval from 0 to 31) and a CSV file ('whichdog_all_annots.csv') that contains the information about the annotations. Each row of the CSV file represents a single annotation, and each column shows:<br> - image_id: ID number of the image.<br> - is_candidate: indicates whether the requested labelling is full (0) or candidate (1).<br> - labeler_id: ID number of the labeler.<br> - time: time employed by the labeler to perform the annotation.<br> - answer: label or set of labels provided by the labeler as annotation.<br> - options: subset of possible labels shown to the labeler.<br> - assignment_id: ID number of the assignment<br> - sequence_point: number that indicates the point of the sequence of images of the assignment in which the annotation was provided.<br> - class: ground truth label of the image.
BibTeX
@inproceedings{benaranmunoz2022whichdog,
  author = {Iker Beñaran-Muñoz and Jerónimo Hernández-González and Aritz Pérez},
  title = {{WhichDog: A crowdsourced dataset including candidate set-based labelling}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.7100697},
}

Loading related content...

Figure