In crowdsourcing, quality control typically relies on the assumption that there is a single "correct" answer for every task. However, many real-world applications—such as rating video quality, aesthetic appeal, or sentiment—are partially subjective. In these cases, disagreement among workers is not necessarily an indicator of poor performance; rather, it may reflect the inherent ambiguity of the item being rated. This paper, presented at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP), introduces a probabilistic framework that explicitly models this item-specific variance to better predict human ratings and identify unreliable workers.
The Problem: Homoskedasticity in Subjective Tasks
Standard crowdsourcing models are often "homoskedastic," meaning they assume the level of disagreement (variance) is constant across all items. If a worker’s rating deviates significantly from the average, they are often flagged as a "spammer."
This approach fails for partially subjective tasks. For instance, when rating the smoothness of a video on a smartphone, workers will likely agree on extremely high-end or low-end hardware. However, for mid-range configurations, opinions naturally vary. Treating this variance as uniform noise leads to two problems: it overfits to noisy data when predicting mean ratings, and it unfairly penalizes workers for valid subjective disagreement on "middling" items.
The Heteroskedastic Probabilistic Model
The authors propose a model where both the expected mean rating ($\mu$) and the expected variance ($\sigma$) are treated as functions of the item's specific attributes. By modeling variance as a signal rather than noise, the system can "expect" higher disagreement on certain items.
Key Components:
- Feature-Based Modeling: The model uses item attributes (e.g., CPU speed, core counts) to predict both the mean and the variance of ratings.
- Worker Reliability ($\theta$): Each worker is assigned a probability of being "reliable." If reliable, their rating is drawn from a Normal distribution defined by the item's features. If unreliable, their rating is drawn from a "spammer" distribution centered at the scale's midpoint.
- Inference Algorithms: The paper introduces NEW, a point-estimate version using Expectation-Maximization (EM), and B-NEW, a Bayesian version using Variational Inference to better handle uncertainty in worker reliability.
Experimental Evaluation
The model was evaluated using a Mobile Video UX Dataset containing 24,132 ratings from 3,241 workers. The task involved rating user experience across 370 different hardware configurations. To test robustness, the authors also collected a "gold standard" of unreliable behavior by asking a specific group of workers to provide ratings without actually watching the videos.
Key Results
The proposed heteroskedastic model demonstrated significant improvements over standard Linear Regression (LR) in several areas:
- Variance and Mean Prediction: As shown in the performance metrics, the model (NEW) significantly outperformed LR in predicting the variance of ratings. Because it accounts for heteroskedasticity, it also achieved a lower Mean Absolute Error (MAE) for the average rating, as it avoids overfitting to high-variance data points.
- Robustness to Spammers: The EM-based model maintained high performance even as the proportion of unreliable workers in the dataset increased.
- Effective Spam Detection: By using contextual penalization—penalizing workers less for disagreeing on subjective items and more for disagreeing on "obvious" ones—the model achieved superior Area Under the Curve (AUC) scores for detecting spammers compared to traditional deviation-based baselines.
Conclusion and Future Work
By treating disagreement as a predictable property of an item rather than a nuisance, this framework provides a more sophisticated approach to crowdsourcing quality control. While the current evaluation focused on mobile hardware UX, the methodology is generalizable to other subjective domains. Future research may expand this model to include "Schools of Thought," allowing the system to cluster workers into groups with different but equally valid subjective preferences, such as "power users" versus "casual users."
Summary updated