TL;DR
This paper explores the potential of crowdsourced workers to assist in diagnosing autism by rating behavioral features in YouTube videos. They found that while crowdsourced workers can achieve accuracy comparable to experts, there's significant variability in their performance, suggesting a need for a robust worker filtration process to ensure reliable data for precision medicine.
What Problem Does This Paper Address?
Diagnosing developmental delays like Autism Spectrum Disorder (ASD) is challenging, with limited access to specialists, especially in rural or lower socioeconomic areas. This often leads to long wait times for diagnosis and treatment. Since ASD is largely a behavioral disorder, video analysis offers a promising avenue for diagnosis. However, manually analyzing vast amounts of video data is time-consuming and expensive. This paper investigates whether crowdsourcing platforms can provide a scalable and affordable solution for collecting behavioral data from videos, which can then be used to train machine learning models for more automated and accessible diagnostics. The core problem is to determine if crowdsourced workers can reliably extract the nuanced behavioral features needed for accurate ASD diagnosis and if their performance is consistent enough for clinical applications.
Methods
The researchers employed a three-pronged approach:
- Clinically Representative Video Set: They curated a dataset of publicly available YouTube videos of children with and without autism. Videos were selected based on clear visibility of the child's face and hands, opportunities for social engagement, and the presence of object interaction.
- Machine Learning Classifier: A pre-validated binary logistic regression classifier, trained on clinician-filled Autism Diagnostic Observation Schedule (ADOS) scoresheets, was used. This classifier takes categorical ordinal behavioral ratings as input and outputs a probability score for autism. The probability of the correct class (PCC) was used as a measure of confidence.
- Crowdsourcing Task: They recruited workers from two platforms: Amazon Mechanical Turk (MTurk) and Microworkers. Workers from distinct geographic locations (Bangladesh, Kenya, Philippines) were targeted on Microworkers to assess cross-cultural performance. Workers watched videos and answered multiple-choice questions about the child's behavior, which were then fed into the logistic regression classifier.
Datasets
- YouTube Videos: A collection of publicly available YouTube videos of children with and without autism was used. The exact number of videos is not explicitly stated for the main study, but the abstract mentions "unstructured public YouTube videos," and the methods section refers to "24 videos" for Microworkers and "43 videos" for MTurk pilot data.
- Clinician-Filled Electronic Health Records: Data from ADOS scoresheets for 1319 children with autism and 70 neurotypical controls were used to train the logistic regression classifier. This serves as the "gold standard" for evaluating worker performance.
Key Insights and Ideas
- "Super Recognizers" Exist: The study identified a subset of workers, termed "super recognizers," who consistently performed at a high level, achieving accuracy comparable to experts. This suggests that specialized skills for behavioral feature recognition can be found within the crowd.
- Variability is the Norm: A significant finding is the wide variation in performance across all worker groups and platforms. This highlights that not all crowd workers are equally adept at this task, necessitating careful selection.
- Geographic and Platform Differences: While the study aimed to compare performance across different countries and platforms, the paper notes that it's unclear whether language barriers, demographic differences, or other factors contribute to the observed inconsistencies.
- Filtration is Crucial: The inherent variability strongly suggests that a robust filtration process is essential before incorporating crowdsourced data into clinical diagnostic pipelines. Simply using raw crowd data is insufficient.
- Time Spent is a Weak Predictor: The correlation between time spent on the task and performance was weak, suggesting that simply ensuring workers spend a minimum amount of time doesn't guarantee quality.
Key Results
- High-Performing Workers: In all four groups (MTurk, Bangladesh, Kenya, Philippines), at least one worker achieved a mean classifier PCC exceeding 75%.
- "Super Recognizer" Threshold: A significant number of workers met the "super recognizer" criteria (mean PCC > 75% and mean PCC > 1 standard deviation above 50%). Specifically, one MTurk worker, three from Bangladesh, two from Kenya, and ten from the Philippines qualified.
- Accuracy on Par with Experts: The best worker responses achieved a mean probability of the correct class above 80%, with over one standard deviation above 50%, indicating accuracy and variability comparable to experts in prior studies.
- Weak Correlation with Time: A weak, though statistically significant, positive correlation (r = 0.358, p = 0.005) was found between mean time spent on task and mean performance across all Microworkers.
Limitations and Future Work
- Video Standardization: The use of unstructured YouTube videos means that video difficulty levels can vary, potentially influencing worker performance. The authors suggest that more structured videos, like those from home-based interventions, might yield more consistent results.
- Demographic Factors: The study acknowledges that further investigation into how demographic factors influence worker performance is needed.
- Replication and Expansion: The authors call for replication studies with different video sets and recruitment strategies. They also propose extending this approach to other pediatric behavioral health conditions like ADHD, speech delay, and OCD.
- Formalized Filtration Process: The paper proposes a novel workflow for recruiting and filtering crowdsourced workers, emphasizing the need for iterative evaluation and progressive filtering to curate a reliable workforce of "super recognizers" for precision medicine applications.
Summary updated