TL;DR
This paper presents a semi-systematic review of crowdsourcing applications in human pathology, revealing that non-expert "crowd workers" can achieve expert-level accuracy in tasks like malaria parasite counting and cancer cell identification. The authors also introduce a novel Bayesian method for literature searching and—fittingly—use crowdsourcing to help screen the papers for their own review.
What Problem Does This Paper Address?
Pathology is facing a dual crisis: a shrinking workforce and a massive data bottleneck. In the UK, academic pathology saw a 60% drop between 2000 and 2012, while the US predicts a 35% decline in the pathologist-to-population ratio by 2030.
While automated machine learning (ML) tools are the long-term solution, they require massive, validated datasets to train—data that currently requires expensive expert time to produce. The authors explore whether crowdsourcing—outsourcing simple but tedious tasks to a large group of untrained individuals—can bridge this gap by providing high-quality data at scale or even assisting in primary diagnosis in resource-poor areas.
Methods
The authors employed a three-pronged methodological approach:
- Bayesian Search Strategy: To avoid the subjectivity of choosing keywords, they used a rigorous probabilistic approach. They created two text corpuses from PubMed: a "P-Corpus" (pathology-related abstracts) and an "N-Corpus" (non-pathology abstracts). Using Bayes' theorem, they ranked words based on how likely they were to indicate a pathology-related paper.
- Meta-Crowdsourcing: After an initial automated search returned 487 unique articles, the authors used the Qualtrics platform to recruit 31 medical students and doctors to screen the abstracts. This "crowdsourced screening" helped filter the list down to 83 candidates for full-text review.
- Systematic Review: The final seven articles that met all criteria (primary research, peer-reviewed, human pathology, and active crowdsourcing) were analyzed for their logistics, crowd demographics, and diagnostic outcomes.
Datasets
The study itself is a review, but it utilized several internal datasets for its methodology:
- P-Corpus: 172,723 abstracts from 2016 containing the term "pathology."
- N-Corpus: 12,000 abstracts from 2016 that did not contain "pathology" (using "university" as a generic anchor term).
- Bayesian Word Lists: A set of 434 candidate words was filtered down to 38 high-probability pathology terms (e.g., "metastasis," "biopsy," "carcinoma") to build the final search expression.
Key Insights and Ideas
- The "Expert" Crowd: The most striking finding is that untrained individuals, when given brief tutorials or gamified tasks, can match or even exceed expert performance. In one study, 6,000 participants counting malaria parasites achieved >99% accuracy.
- Gamification is a Powerful Incentive: Several successful studies used "biogames" to recruit thousands of participants. This suggests that making pathology tasks entertaining can solve the recruitment problem without needing high monetary incentives.
- The Filtering Pipeline: A promising model for the future is using the crowd as a "pre-filter." In resource-poor areas, a large crowd could screen thousands of red blood cell images, leaving only the most ambiguous or difficult cases for the few available expert pathologists.
- Exponential Growth: The authors noted a three-fold increase in crowdsourcing-related health research between 2013 and 2016, signaling that this field is rapidly maturing.
Key Results
The review synthesized findings from seven core papers covering microbiology, molecular pathology, in-vivo microscopy, and immunohistochemistry (IHC):
- Malaria Diagnosis: Studies showed that crowdsourced gamers could identify infected cells with 98.13% accuracy compared to ground truth.
- Cancer Classification: In a study of bladder cancer using in-vivo microscopy, the crowd achieved 92% diagnostic accuracy.
- IHC Scoring: Crowdsourcing was found to be a viable method for scoring protein expression in tumor images, often showing better concordance with pathologists than existing automated computational methods at the time.
- Scale: Participant numbers ranged from 31 to nearly 100,000, demonstrating the massive scalability of the approach.
Limitations and Future Work
Despite the promising results, the authors highlight several hurdles:
- Privacy and Ethics: Using the general public to view patient images raises significant concerns regarding patient privacy and data security that have not yet been fully addressed.
- Demographic Blind Spots: Most studies failed to report the demographics (age, education, language) of their crowd, which could significantly impact the reliability and bias of the results.
- The AI Threshold: As computer vision improves, there is a looming question: when will AI surpass the crowd? The authors suggest that even if AI wins on accuracy, crowdsourcing will remain valuable for generating training data and increasing public awareness of health issues (e.g., breast cancer).
- Complexity Limits: Tasks requiring highly specialized clinical judgment cannot yet be crowdsourced; identifying the "threshold of complexity" remains a key area for future research.
Summary updated