Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder

Peter Washington, Émilie Leblanc, Kaitlyn Dunlap, Yordan Penev, Maya Varma, Jae-Yoon Jung, Brianna Sierra Chrisman, Min Woo Sun, Nate Tyler Stockham, Kelley Marie Paskov, Haik Kalantarian, Catalin Voss, Nick Haber, Dennis P. Wall

TL;DR

This paper introduces a novel set of behavioral metrics to efficiently and reliably identify trustworthy crowd workers for labeling video data of children with autism. By analyzing worker consistency and engagement, these metrics can predict worker performance with high accuracy, paving the way for scalable and affordable telemedical diagnostic tools.

What Problem Does This Paper Address?

The increasing prevalence of autism spectrum disorder (ASD) coupled with a shortage of diagnostic resources, particularly in underserved areas, highlights the need for accessible diagnostic solutions. Telemedicine offers a promising avenue, but leveraging remote data collection and analysis, especially for sensitive health information, requires a robust system for ensuring the quality and trustworthiness of human annotators. Traditional methods of credentialing crowd workers are time-consuming and expensive. This paper addresses the challenge of efficiently and reliably selecting trustworthy crowd workers for the crucial task of labeling behavioral features in videos of children with ASD, without compromising patient privacy or requiring extensive human subject training for each worker.

Methods

The authors developed and evaluated several metrics to assess the trustworthiness and reliability of crowd workers. These metrics are designed to be derived from worker behavior during a labeling task, without requiring ground truth labels of the child's diagnosis.

  • Mean Same-Child L1 Distance (MSCL₁): Measures test-retest reliability by calculating the difference in a worker's answers for the same video when rated a month apart. Higher values indicate greater variability.
  • Mean Pairwise Internal L1 Distance (MPIL₁): Assesses consistency across different videos by measuring the average difference in a worker's answer vectors for all pairs of videos they rated. Higher values suggest more diverse answer patterns.
  • Penalized Time (PT): A composite metric that rewards workers who spend sufficient time on the initial rating and also considers their response to a warning about insufficient time, incorporating both initial rating time and revision time, as well as the change in answers after revision.
  • Time Spent: The raw duration a worker spends rating each video.

These metrics were then used as input features for a linear regression model to predict the worker's performance, measured by the probability of the correct class (PCC) when their labels are fed into a pre-validated logistic regression classifier for autism diagnosis.

Datasets

The study utilized a dataset of 24 publicly available YouTube videos of children with autism and matched neurotypical controls. These videos were selected based on specific criteria, including the visibility of the child's face and hands, opportunities for social engagement, and the presence of object interaction. The videos were short, with a mean duration of 47.75 seconds.

Key Insights and Ideas

  • Behavioral metrics are powerful predictors: The study demonstrates that simple behavioral metrics derived from how crowd workers perform a task can be highly predictive of their performance quality, even without knowing the true diagnosis of the individuals in the videos.
  • Consistency is key: The "Mean Pairwise Internal L1 Distance" (MPIL₁) emerged as the most predictive metric, suggesting that workers who exhibit a greater diversity in their answers across different videos are more likely to be accurate. This counter-intuitive finding might stem from the inherent variability in behavioral features across children with ASD.
  • Time alone is not enough: While time spent on a task is a common indicator of effort, this study found that raw time spent was not as predictive as consistency metrics. Furthermore, workers who were warned about spending too little time did not necessarily improve their performance, suggesting they might simply keep the interface open without genuine engagement.
  • Privacy-preserving evaluation: The methodology allows for the evaluation of crowd worker trustworthiness without exposing sensitive patient data during the evaluation phase, which is crucial for real-world applications.

Key Results

  • The MPIL₁ metric alone could predict a worker's PCC with a Mean Average Error (MAE) of 6.98% when trained and tested on the same set of workers but different videos, and 10.91% when tested on an entirely distinct set of workers and videos.
  • A linear regression model using a combination of MSCL₁ and MPIL₁ achieved an MAE of 10.41% for predicting PCC on a distinct set of videos, demonstrating good generalization.
  • The study found that workers with consistently low performance (PCC below 50%) tended to not re-rate videos or revise their answers, highlighting the utility of the developed metrics in identifying unreliable workers.
  • Penalized time (PT) and mean time spent were not significantly correlated with PCC, suggesting that these metrics are less effective on their own.

Limitations and Future Work

  • Limited dataset size: The study used a relatively small dataset of 24 videos, which might limit the generalizability of the findings.
  • Crowdsourcing platform specific: The study utilized Microworkers.com, and the findings might not directly translate to other crowdsourcing platforms with different worker pools or functionalities.
  • Future work: The authors suggest evaluating workers on a larger scale to validate the preliminary findings. They also propose exploring predictive time-based trustworthiness metrics and incorporating more fine-grained timing information to better understand worker engagement. Further research could also focus on developing more precise prediction models by combining these metrics with other potential indicators of worker reliability.

Summary updated

BibTeX
@inproceedings{washington2021selection,
  author = {Peter Washington and Émilie Leblanc and Kaitlyn Dunlap and Yordan Penev and Maya Varma and Jae-Yoon Jung and Brianna Sierra Chrisman and Min Woo Sun and Nate Tyler Stockham and Kelley Marie Paskov and Haik Kalantarian and Catalin Voss and Nick Haber and Dennis P. Wall},
  title = {{Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder}},
  booktitle = {PSB},
  year = {2021},
  doi = {10.1142/9789811232701\_0002},
}

Loading related content...

Figure