Do you see what I see: crowdsource annotation of captured scenes

J. Aaron Hipp, Deepti Adlakha, Rebecca Gernes, Agata Kargol, Robert PlessSenseCam 2013 · DOI

TL;DR

This paper explores using crowdsourcing platforms like Amazon Mechanical Turk (MTurk) to annotate large datasets of outdoor webcam images, specifically for counting pedestrians, cyclists, and vehicles. They found that MTurk workers can provide reliable and valid annotations, with the average of four workers being an optimal threshold for accuracy and cost-effectiveness. This work demonstrates the potential of crowdsourcing for large-scale behavioral analysis using readily available visual data.

What Problem Does This Paper Address?

Analyzing real-world human behavior, particularly physical activity and transportation patterns, often requires annotating vast amounts of visual data. Traditional annotation methods, relying on trained research assistants, are expensive and time-consuming. The authors address the challenge of efficiently and affordably acquiring these annotations by investigating the viability of crowdsourcing platforms. Specifically, they aim to determine if crowdsourced annotations from non-expert workers can achieve sufficient reliability and validity compared to trained annotators, using images from the Archive of Many Outdoor Scenes (AMOS) dataset, which captures street-level views.

Methods

The researchers utilized images from six AMOS webcams located in Washington D.C., capturing scenes before and after built environment changes (e.g., new bicycle lanes). A total of 1,440 images were collected, with 240 images per camera, taken between 7:00 am and 7:00 pm during the first week of June in 2007 (pre-change) and 2010 (post-change).

These images were then annotated by Amazon Mechanical Turk (MTurk) workers. Each image was annotated by five unique MTurk workers, who were tasked with counting pedestrians, cyclists, and vehicles. The MTurk workers were paid $0.01 per scene and took an average of 71 seconds per task. For comparison and validation, two trained research assistants also annotated the same set of images.

The study employed statistical methods to assess reliability and validity. Inter-rater reliability (IRR) was calculated using Pearson's R, Inter-Class Correlations (ICCs), and Cohen's Kappa. These statistics were computed for various combinations of MTurk workers (two, three, four, and five) to determine the optimal number of workers needed for reliable results. The validity of MTurk annotations was assessed by comparing them against the annotations provided by the trained research assistants.

Datasets

  • Archive of Many Outdoor Scenes (AMOS) dataset: This is a large collection of images from publicly available outdoor webcams. For this study, a subset of 1,440 images from six specific webcams in Washington D.C. was used, capturing scenes before (2007) and after (2010) built environment improvements.

Key Insights and Ideas

  • Crowdsourcing as a viable annotation workforce: The paper strongly suggests that crowdsourcing platforms like MTurk can be a reliable and valid method for annotating outdoor scenes, particularly for common objects like pedestrians and vehicles.
  • Optimal number of workers: The study found that using the average annotation of four unique MTurk workers provided the most parsimonious balance between reliability and cost. While more workers generally increased reliability, the gains diminished significantly after four workers.
  • Differential reliability for object types: Annotations for pedestrians and vehicles showed substantial agreement among MTurk workers and high validity when compared to trained assistants. However, cyclists, being a rarer event in the captured scenes (present in only about 10% of images), exhibited lower reliability and validity.
  • Cost-effectiveness: The research highlights the significant cost savings of using crowdsourcing. Annotating 7,200 images (5 workers x 1,440 images) cost only $72.00, making it a highly economical option for large-scale data annotation.
  • Potential for behavioral analysis: The findings open doors for using readily available webcam data and crowdsourcing to study physical activity and transportation behaviors on a much larger scale than previously feasible.

Key Results

  • Pedestrian reliability: The average reliability (Pearson's R) for pedestrians increased from 0.562 with two MTurk workers to 0.848 with five workers. Trained RAs achieved an average of 0.850.
  • Vehicle reliability: For vehicles, the average reliability (Pearson's R) ranged from 0.354 with two MTurk workers to 0.705 with five workers. Trained RAs achieved a higher average of 0.885.
  • Cyclist reliability: Cyclist reliability was generally lower, with averages ranging from 0.333 (two workers) to 0.645 (five workers). Trained RAs had an average of 0.329.
  • Validity vs. RAs: The validity statistics (Pearson's R) between MTurk workers and trained RAs were high for pedestrians (0.846 – 0.901) and vehicles (0.753 – 0.857), but only fair to moderate for cyclists (0.361 – 0.494).
  • Optimal threshold: Taking the mean annotation of four MTurk workers was identified as the most parsimonious threshold for valid results, achieving substantial agreement (scores between 0.600-0.799).
  • No significant difference post-BE change: The results for images captured after the built environment changes (2010) were not statistically different from those captured before the changes (2007).

Limitations and Future Work

  • Cyclist annotation challenges: The paper acknowledges that cyclists, due to their infrequent appearance in many scenes, pose a challenge for crowdsourced annotation. MTurk is not yet ideal for capturing rare events without additional instructions or worker limitations.
  • Future work: The authors suggest that this research is a novel demonstration of unobtrusive surveillance for evaluating physical activity and transportation behavior. Future work could explore refining MTurk tasks for rare events, potentially by providing more specific training or filtering workers, and expanding the types of behaviors and objects that can be annotated using this methodology.

Summary updated

BibTeX
@inproceedings{hipp2013do,
  author = {J. Aaron Hipp and Deepti Adlakha and Rebecca Gernes and Agata Kargol and Robert Pless},
  title = {{Do you see what I see: crowdsource annotation of captured scenes}},
  booktitle = {SenseCam},
  year = {2013},
  doi = {10.1145/2526667.2526671},
}
Figure