Abstract
Many research domains use data elicited from "citizen scientists" when a direct measure of a process is expensive or infeasible. However, participants may report incorrect estimates or classifications due to their lack of skill. We demonstrate how Bayesian hierarchical models can be used to learn about latent variables of interest, while accounting for the participants' abilities. The model is described in the context of an ecological application that involves crowdsourced classifications of georeferenced coral-reef images from the Great Barrier Reef, Australia. The latent variable of interest is the proportion of coral cover, which is a common indicator of coral reef health. The participants' abilities are expressed in terms of sensitivity and specificity of a correctly classified set of points on the images. The model also incorporates a spatial component, which allows prediction of the latent variable in locations that have not been surveyed. We show that the model outperforms traditional weighted-regression approaches used to account for uncertainty in citizen science data. Our approach produces more accurate regression coefficients and provides a better characterization of the latent process of interest. This new method is implemented in the probabilistic programming language Stan and can be applied to a wide number of problems that rely on uncertain citizen science data.
Introduction: The Citizen Science Dilemma
Crowdsourcing platforms like Zooniverse and iNaturalist have revolutionized ecological monitoring, enabling researchers to process thousands of images that would be impossible for small expert teams to handle. However, "citizen scientists" vary significantly in skill. In coral reef monitoring, for instance, a novice might easily mistake complex algae structures for hard coral, leading to systematic errors.
Standard statistical approaches often either ignore these errors—leading to biased results—or use simple weighted models based on overall accuracy. Furthermore, ecological data is inherently spatial; conditions at one reef location are typically related to those nearby. Most existing models fail to combine error correction with this crucial spatial context, resulting in poor predictions for unsurveyed areas.
The SDME Model: Correcting for Human Error and Spatial Context
To address these challenges, the authors propose the Spatially Dependent Misclassification Error (SDME) model. Implemented in the probabilistic programming language Stan, this Bayesian hierarchical framework features:
- Latent Variable Modeling: The "true" coral cover is treated as a hidden (latent) variable observed through the noisy lens of participant classifications.
- Sensitivity and Specificity: Rather than a single accuracy score, the model estimates individual sensitivity (the ability to correctly identify coral) and specificity (the ability to correctly identify "not coral"). This helps mitigate the "algae trap," where novices misclassify algae as coral.
- Spatial Autocorrelation: Using a Conditional Autoregressive (CAR) prior, the model divides the study area into Voronoi polygons. This allows the model to "borrow strength" from neighboring locations to improve estimates and predict coral cover in "blue zones" where no images were taken.
Datasets and Validation
The study utilized a combination of simulated and real-world data from the Great Barrier Reef (GBR):
- GBR Dataset: 1,585 georeferenced images from the XL Catlin Seaview Survey.
- Expert "Gold Standard": 171 images classified by professional marine scientists to calibrate the model.
- Crowdsourced Data: 343 images classified by 157 workers on Amazon Mechanical Turk.
- Simulation Study: 300 datasets used to verify that the model could accurately recover known parameters compared to simpler models.
Results: Outperforming Traditional Approaches
In simulations, the SDME model significantly outperformed traditional weighted-regression approaches. It accurately captured the true effect of environmental predictors, such as thermal stress, whereas weighted models produced narrow, biased estimates. The posterior distributions of latent coral cover fell within the 95% highest density interval for 97.78% of the simulated cells.
When applied to the GBR case study, the model confirmed that Degree Heating Weeks (thermal stress) and cyclone exposure have substantial negative effects on coral cover. Conversely, no-take marine reserves showed a clear positive correlation with reef health, demonstrating the model's utility for ecological validation.
Conclusion and Future Directions
While the SDME model provides a robust framework for correcting crowdsourced data, it is computationally intensive due to the nature of Bayesian MCMC simulations. Future work could address scalability for millions of participants and implement mixture models to account for varying task difficulty across different images. Additionally, extending the model to include temporal dimensions could create a dynamic "early warning system" for reef bleaching events globally.
Summary updated