Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

Peter Washington, Émilie Leblanc, Kaitlyn Dunlap, Yordan Penev, Aaron Kline, Kelley Paskov, Min Sun, Brianna Chrisman, Nate Stockham, Maya Varma, Catalin Voss, Nick Haber, Dennis P. WallJournal of Personalized Medicine 2020 · DOI

TL;DR

This paper explores the potential of crowdsourced workers to assist in diagnosing autism by rating behavioral features in YouTube videos. They found that while crowdsourced workers can achieve accuracy comparable to experts, there's significant variability in their performance, suggesting a need for a robust worker filtration process to ensure reliable data for precision medicine.

What Problem Does This Paper Address?

Diagnosing developmental delays like Autism Spectrum Disorder (ASD) is challenging, with limited access to specialists, especially in rural or lower socioeconomic areas. This often leads to long wait times for diagnosis and treatment. Since ASD is largely a behavioral disorder, video analysis offers a promising avenue for diagnosis. However, manually analyzing vast amounts of video data is time-consuming and expensive. This paper investigates whether crowdsourcing platforms can provide a scalable and affordable solution for collecting behavioral data from videos, which can then be used to train machine learning models for more automated and accessible diagnostics. The core problem is to determine if crowdsourced workers can reliably extract the nuanced behavioral features needed for accurate ASD diagnosis and if their performance is consistent enough for clinical applications.

Methods

The researchers employed a three-pronged approach:

  1. Clinically Representative Video Set: They curated a dataset of publicly available YouTube videos of children with and without autism. Videos were selected based on clear visibility of the child's face and hands, opportunities for social engagement, and the presence of object interaction.
  2. Machine Learning Classifier: A pre-validated binary logistic regression classifier, trained on clinician-filled Autism Diagnostic Observation Schedule (ADOS) scoresheets, was used. This classifier takes categorical ordinal behavioral ratings as input and outputs a probability score for autism. The probability of the correct class (PCC) was used as a measure of confidence.
  3. Crowdsourcing Task: They recruited workers from two platforms: Amazon Mechanical Turk (MTurk) and Microworkers. Workers from distinct geographic locations (Bangladesh, Kenya, Philippines) were targeted on Microworkers to assess cross-cultural performance. Workers watched videos and answered multiple-choice questions about the child's behavior, which were then fed into the logistic regression classifier.

Datasets

  • YouTube Videos: A collection of publicly available YouTube videos of children with and without autism was used. The exact number of videos is not explicitly stated for the main study, but the abstract mentions "unstructured public YouTube videos," and the methods section refers to "24 videos" for Microworkers and "43 videos" for MTurk pilot data.
  • Clinician-Filled Electronic Health Records: Data from ADOS scoresheets for 1319 children with autism and 70 neurotypical controls were used to train the logistic regression classifier. This serves as the "gold standard" for evaluating worker performance.

Key Insights and Ideas

  • "Super Recognizers" Exist: The study identified a subset of workers, termed "super recognizers," who consistently performed at a high level, achieving accuracy comparable to experts. This suggests that specialized skills for behavioral feature recognition can be found within the crowd.
  • Variability is the Norm: A significant finding is the wide variation in performance across all worker groups and platforms. This highlights that not all crowd workers are equally adept at this task, necessitating careful selection.
  • Geographic and Platform Differences: While the study aimed to compare performance across different countries and platforms, the paper notes that it's unclear whether language barriers, demographic differences, or other factors contribute to the observed inconsistencies.
  • Filtration is Crucial: The inherent variability strongly suggests that a robust filtration process is essential before incorporating crowdsourced data into clinical diagnostic pipelines. Simply using raw crowd data is insufficient.
  • Time Spent is a Weak Predictor: The correlation between time spent on the task and performance was weak, suggesting that simply ensuring workers spend a minimum amount of time doesn't guarantee quality.

Key Results

  • High-Performing Workers: In all four groups (MTurk, Bangladesh, Kenya, Philippines), at least one worker achieved a mean classifier PCC exceeding 75%.
  • "Super Recognizer" Threshold: A significant number of workers met the "super recognizer" criteria (mean PCC > 75% and mean PCC > 1 standard deviation above 50%). Specifically, one MTurk worker, three from Bangladesh, two from Kenya, and ten from the Philippines qualified.
  • Accuracy on Par with Experts: The best worker responses achieved a mean probability of the correct class above 80%, with over one standard deviation above 50%, indicating accuracy and variability comparable to experts in prior studies.
  • Weak Correlation with Time: A weak, though statistically significant, positive correlation (r = 0.358, p = 0.005) was found between mean time spent on task and mean performance across all Microworkers.

Limitations and Future Work

  • Video Standardization: The use of unstructured YouTube videos means that video difficulty levels can vary, potentially influencing worker performance. The authors suggest that more structured videos, like those from home-based interventions, might yield more consistent results.
  • Demographic Factors: The study acknowledges that further investigation into how demographic factors influence worker performance is needed.
  • Replication and Expansion: The authors call for replication studies with different video sets and recruitment strategies. They also propose extending this approach to other pediatric behavioral health conditions like ADHD, speech delay, and OCD.
  • Formalized Filtration Process: The paper proposes a novel workflow for recruiting and filtering crowdsourced workers, emphasizing the need for iterative evaluation and progressive filtering to curate a reliable workforce of "super recognizers" for precision medicine applications.

Summary updated

Abstract

Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted members of popular crowdsourcing platforms-to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (<i>r</i> = 0.358, <i>p</i> = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.
BibTeX
@article{washington2020precision,
  author = {Peter Washington and Émilie Leblanc and Kaitlyn Dunlap and Yordan Penev and Aaron Kline and Kelley Paskov and Min Sun and Brianna Chrisman and Nate Stockham and Maya Varma and Catalin Voss and Nick Haber and Dennis P. Wall},
  title = {{Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition}},
  journal = {Journal of Personalized Medicine},
  year = {2020},
  doi = {10.3390/jpm10030086},
}
Figure