A Review on the Applications of Crowdsourcing in Human Pathology

Roshanak Alialy, Sasan Tavakkol, Amir Ghorbani-Aghbologhi, Alireza Ghaffarieh, Seon Ho Kim, Cyrus Shahabi

TL;DR

This paper presents a semi-systematic review of crowdsourcing applications in human pathology, revealing that non-expert "crowd workers" can achieve expert-level accuracy in tasks like malaria parasite counting and cancer cell identification. The authors also introduce a novel Bayesian method for literature searching and—fittingly—use crowdsourcing to help screen the papers for their own review.

What Problem Does This Paper Address?

Pathology is facing a dual crisis: a shrinking workforce and a massive data bottleneck. In the UK, academic pathology saw a 60% drop between 2000 and 2012, while the US predicts a 35% decline in the pathologist-to-population ratio by 2030.

While automated machine learning (ML) tools are the long-term solution, they require massive, validated datasets to train—data that currently requires expensive expert time to produce. The authors explore whether crowdsourcing—outsourcing simple but tedious tasks to a large group of untrained individuals—can bridge this gap by providing high-quality data at scale or even assisting in primary diagnosis in resource-poor areas.

Methods

The authors employed a three-pronged methodological approach:

  1. Bayesian Search Strategy: To avoid the subjectivity of choosing keywords, they used a rigorous probabilistic approach. They created two text corpuses from PubMed: a "P-Corpus" (pathology-related abstracts) and an "N-Corpus" (non-pathology abstracts). Using Bayes' theorem, they ranked words based on how likely they were to indicate a pathology-related paper.
  2. Meta-Crowdsourcing: After an initial automated search returned 487 unique articles, the authors used the Qualtrics platform to recruit 31 medical students and doctors to screen the abstracts. This "crowdsourced screening" helped filter the list down to 83 candidates for full-text review.
  3. Systematic Review: The final seven articles that met all criteria (primary research, peer-reviewed, human pathology, and active crowdsourcing) were analyzed for their logistics, crowd demographics, and diagnostic outcomes.
The authors used Bayesian probability to identify words most likely to appear in pathology literature, visualized here as a word cloud.

Datasets

The study itself is a review, but it utilized several internal datasets for its methodology:

  • P-Corpus: 172,723 abstracts from 2016 containing the term "pathology."
  • N-Corpus: 12,000 abstracts from 2016 that did not contain "pathology" (using "university" as a generic anchor term).
  • Bayesian Word Lists: A set of 434 candidate words was filtered down to 38 high-probability pathology terms (e.g., "metastasis," "biopsy," "carcinoma") to build the final search expression.
Table 1 shows the most frequent words in the P-Corpus and N-Corpus, illustrating why simple frequency isn't enough to define a search—common words like "pmid" and "author" appear in both.

Key Insights and Ideas

  • The "Expert" Crowd: The most striking finding is that untrained individuals, when given brief tutorials or gamified tasks, can match or even exceed expert performance. In one study, 6,000 participants counting malaria parasites achieved >99% accuracy.
  • Gamification is a Powerful Incentive: Several successful studies used "biogames" to recruit thousands of participants. This suggests that making pathology tasks entertaining can solve the recruitment problem without needing high monetary incentives.
  • The Filtering Pipeline: A promising model for the future is using the crowd as a "pre-filter." In resource-poor areas, a large crowd could screen thousands of red blood cell images, leaving only the most ambiguous or difficult cases for the few available expert pathologists.
  • Exponential Growth: The authors noted a three-fold increase in crowdsourcing-related health research between 2013 and 2016, signaling that this field is rapidly maturing.
Research into crowdsourcing and citizen science has seen exponential growth over the last decade across major medical databases.

Key Results

The review synthesized findings from seven core papers covering microbiology, molecular pathology, in-vivo microscopy, and immunohistochemistry (IHC):

  • Malaria Diagnosis: Studies showed that crowdsourced gamers could identify infected cells with 98.13% accuracy compared to ground truth.
  • Cancer Classification: In a study of bladder cancer using in-vivo microscopy, the crowd achieved 92% diagnostic accuracy.
  • IHC Scoring: Crowdsourcing was found to be a viable method for scoring protein expression in tumor images, often showing better concordance with pathologists than existing automated computational methods at the time.
  • Scale: Participant numbers ranged from 31 to nearly 100,000, demonstrating the massive scalability of the approach.

Limitations and Future Work

Despite the promising results, the authors highlight several hurdles:

  • Privacy and Ethics: Using the general public to view patient images raises significant concerns regarding patient privacy and data security that have not yet been fully addressed.
  • Demographic Blind Spots: Most studies failed to report the demographics (age, education, language) of their crowd, which could significantly impact the reliability and bias of the results.
  • The AI Threshold: As computer vision improves, there is a looming question: when will AI surpass the crowd? The authors suggest that even if AI wins on accuracy, crowdsourcing will remain valuable for generating training data and increasing public awareness of health issues (e.g., breast cancer).
  • Complexity Limits: Tasks requiring highly specialized clinical judgment cannot yet be crowdsourced; identifying the "threshold of complexity" remains a key area for future research.

Summary updated

Abstract

The advent of the digital pathology has introduced new avenues of diagnostic medicine. Among them, crowdsourcing has attracted researchers' attention in the recent years, allowing them to engage thousands of untrained individuals in research and diagnosis. While there exist several articles in this regard, prior works have not collectively documented them. We, therefore, aim to review the applications of crowdsourcing in human pathology in a semi-systematic manner. We first, introduce a novel method to do a systematic search of the literature. Utilizing this method, we, then, collect hundreds of articles and screen them against a predefined set of criteria. Furthermore, we crowdsource part of the screening process, to examine another potential application of crowdsourcing. Finally, we review the selected articles and characterize the prior uses of crowdsourcing in pathology.
BibTeX
@article{alialy2018review,
  author = {Roshanak Alialy and Sasan Tavakkol and Amir Ghorbani-Aghbologhi and Alireza Ghaffarieh and Seon Ho Kim and Cyrus Shahabi and Cyrus Shahabi},
  title = {{A Review on the Applications of Crowdsourcing in Human Pathology}},
  journal = {Journal of Pathology Informatics},
  year = {2018},
  doi = {10.4103/jpi.jpi\_65\_17},
}
Figure