Abstract
Crowdsourcing is a powerful tool for scaling data labeling, yet aggregating complex "item ordering" responses—such as ranking sentences in a paragraph—remains a significant challenge. Traditional quality control methods like majority voting fail in high-dimensional permutation spaces where identical responses are rare. This paper introduces a probabilistic model that treats worker responses as points in a vector space, weighting inputs by estimated worker "ability." Using Spearman distance and Maximum Likelihood Estimation, the model iteratively refines the estimated true ordering and worker reliability. Experimental results on word and sentence ordering tasks demonstrate that this approach consistently outperforms baseline aggregation, particularly as the number of workers per task increases.
Introduction
In crowdsourcing, quality control is typically optimized for binary or multiple-choice tasks. Methods like the Dawid-Skene model effectively identify reliable workers for simple labels. However, item ordering tasks—where $N$ items must be arranged in a specific sequence—present a much larger search space. For a set of 6 items, there are 720 possible permutations ($N!$), making it highly unlikely for multiple workers to provide the same incorrect ordering by chance. Consequently, simple majority voting is insufficient. This research addresses the need for a statistical framework capable of estimating ground truth from noisy, unreliable crowd rankings while simultaneously quantifying worker trustworthiness.
Methods
The proposed approach utilizes a generative model based on distance-based ordering. The core assumption is that a worker's response is a noisy observation of the true order, with the probability of a specific ranking decreasing as its distance from the truth increases.
The Probabilistic Model
The model incorporates a worker ability parameter, $\lambda^{(k)}$, for each worker $k$. A high $\lambda$ indicates an expert whose responses cluster tightly around the ground truth, while a low $\lambda$ indicates a worker whose responses are closer to random noise. By employing the Spearman distance (the Euclidean distance between rank vectors), the researchers treat rankings as coordinates in a high-dimensional vector space.
Estimation Algorithm
To determine the true ordering and worker abilities, the authors use an iterative Maximum Likelihood Estimation (MLE) algorithm:
- Step A: Given current estimates of worker abilities, calculate the "weighted mean" of the ranks to estimate the true order.
- Step B: Given the estimated true order, optimize the worker ability parameters based on the distance of their individual responses from that truth.
The model was evaluated using two real-world datasets from the Lancers marketplace: word ordering (arranging words into sentences) and sentence ordering (restoring logical flow to a paragraph).
Results
The researchers compared their distance-based model against a baseline that aggregates rankings without weighting for worker ability.
Performance and Redundancy
A key finding is the "redundancy threshold." The sophisticated quality control model begins to significantly outperform the baseline once the number of workers per task exceeds 5 or 6. Below this threshold, the data is often too sparse for the "intelligence" of the model to provide a meaningful advantage.
Accuracy Gains
As shown in the experimental results, the proposed method consistently achieves a lower Averaged Squared Error (Spearman distance from ground truth) compared to the baseline. The model effectively filters noise in both word-level and sentence-level tasks, demonstrating the versatility of the distance-based approach across different ordering problems. By treating ability as a continuous parameter rather than a binary classification, the model provides a more nuanced weighting of every contribution.
Discussion and Conclusion
While the model is effective for small sets (5-6 items), the factorial growth of the search space ($N!$) presents computational challenges for larger tasks. Future work may require heuristic search methods to handle dozens of items. Additionally, the current model assumes static worker ability; incorporating temporal changes or task-specific expertise could further improve accuracy. Finally, exploring alternative distance metrics like Kendall’s Tau might better capture specific human error patterns. Despite these limitations, the framework provides a robust solution for aggregating complex crowd-sourced rankings.
Summary updated