TL;DR
This paper explores using crowdsourcing platforms like Amazon Mechanical Turk to screen citations for systematic reviews, aiming to reduce the high cost and time involved. They found that crowdsourced screening, when aggregated using various algorithms, can achieve high accuracy in identifying relevant studies while significantly cutting costs and speeding up the process compared to traditional expert review.
What Problem Does This Paper Address?
Systematic reviews are crucial for evidence-based healthcare, but their creation is incredibly time-consuming and expensive. A major bottleneck is the initial "citation screening" phase, where researchers sift through thousands of titles and abstracts to find a small subset of relevant studies. The sheer volume of published literature makes this task increasingly burdensome, hindering timely access to evidence for decision-making. This paper investigates whether crowdsourcing, leveraging untrained online workers, can offer a viable solution to this problem.
Methods
The core of the study involved using Amazon Mechanical Turk (MTurk) to have untrained crowdworkers screen citations from four existing systematic reviews. Instead of asking workers for a simple "include/exclude" decision, the researchers broke down the eligibility criteria into a series of 4-5 simple, pattern-matching questions for each citation. This approach aimed to simplify the task for laypeople and reduce the need for deep medical expertise.
After collecting responses from multiple workers (typically 5) for each citation, the researchers employed nine different aggregation algorithms to combine these individual decisions into a final "include" or "exclude" judgment. These algorithms ranged from very inclusive (e.g., including a citation if any worker flagged it) to more conservative approaches. The performance of these crowdsourced decisions was then compared against the gold standard of decisions made by trained experts. Quality control was implemented through qualification tests and "honeypots" (hidden control questions) to identify and filter out low-quality or unconscientious workers.
Datasets
The study utilized citation data from four previously conducted systematic reviews:
- Proton beam (pilot review): Focused on the comparative effectiveness and safety of charged-particle radiation therapies for cancer.
- Appendicitis: Investigated the diagnostic performance of tests for suspected acute appendicitis.
- Decision Support Tools (DST): Examined the effectiveness of decision aids for people facing cancer screening and treatment decisions.
- Omega-3: Assessed the association of omega-3 fatty acid intake with cardiovascular disease risk factors and outcomes.
These datasets were chosen to represent a variety of medical topics and question types, aiming to test the generalizability of the crowdsourcing approach.
Key Insights and Ideas
- Decomposition is Key: Breaking down complex eligibility criteria into simple, sequential questions was crucial for enabling untrained workers to participate effectively. This "artificial artificial intelligence" approach bypasses the need for domain expertise.
- Aggregation Matters: The choice of aggregation algorithm significantly impacts the trade-off between identifying all relevant studies (yield) and excluding irrelevant ones (gain). More inclusive algorithms achieved higher yield but lower gain, and vice-versa.
- Quality Control is Essential: While crowdsourcing offers cost and speed benefits, robust quality control mechanisms (qualification tests and honeypots) are vital to filter out unreliable workers and ensure data quality.
- Cost and Time Savings are Substantial: Crowdsourcing demonstrated a dramatic reduction in both the time required for screening and the overall cost compared to traditional expert review.
Key Results
- High Yield: The most inclusive aggregation strategy (including a citation if any worker flagged it) achieved a yield of 95% to 99%, meaning it captured almost all relevant studies.
- Significant Gain: This same inclusive strategy excluded 68% to 82% of irrelevant citations. Other algorithms could increase this "gain" (better exclusion of irrelevant papers) at the cost of slightly lower yield.
- Cost Reduction: Crowdsourced screening cost between $460 and $2220, representing a cost reduction of up to 88% compared to trained experts.
- Speed: Screening was completed in 4 to 17 days, a significant acceleration compared to the months often required for expert screening.
- Worker Agreement: Fleiss Kappa scores indicated moderate agreement among workers on individual questions, highlighting the need for effective aggregation strategies.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Dataset Size: The study used only four systematic review datasets, and further replication across diverse research areas is needed to confirm generalizability.
- Convenience Sample: Citations were selected from specific phases or questions of reviews, and the impact of this selection bias is unknown.
- Sequential Learning: The researchers learned and refined their methods through iterative experiments, meaning later datasets benefited from earlier lessons.
Future work suggested includes:
- Replication: Repeating the study with more datasets and in different research domains.
- Enhanced Quality Control: Developing more sophisticated methods to identify and exclude problematic workers, potentially through "streaker" detection or hybrid human-machine approaches.
- Hybrid Models: Combining crowdsourcing with text-classification algorithms to further automate screening and improve sensitivity.
- Ethical Considerations: Addressing fair worker compensation and ensuring that crowdsourcing does not lead to exploitation, while also noting the positive feedback received from workers.
Summary updated