Crowdsourcing Quality Control for Item Ordering Tasks

Toshiko Matsui, Yukino Baba, Toshihiro Kamishima, Hisashi Kashima

Abstract

Crowdsourcing is a powerful tool for scaling data labeling, yet aggregating complex "item ordering" responses—such as ranking sentences in a paragraph—remains a significant challenge. Traditional quality control methods like majority voting fail in high-dimensional permutation spaces where identical responses are rare. This paper introduces a probabilistic model that treats worker responses as points in a vector space, weighting inputs by estimated worker "ability." Using Spearman distance and Maximum Likelihood Estimation, the model iteratively refines the estimated true ordering and worker reliability. Experimental results on word and sentence ordering tasks demonstrate that this approach consistently outperforms baseline aggregation, particularly as the number of workers per task increases.

Introduction

In crowdsourcing, quality control is typically optimized for binary or multiple-choice tasks. Methods like the Dawid-Skene model effectively identify reliable workers for simple labels. However, item ordering tasks—where $N$ items must be arranged in a specific sequence—present a much larger search space. For a set of 6 items, there are 720 possible permutations ($N!$), making it highly unlikely for multiple workers to provide the same incorrect ordering by chance. Consequently, simple majority voting is insufficient. This research addresses the need for a statistical framework capable of estimating ground truth from noisy, unreliable crowd rankings while simultaneously quantifying worker trustworthiness.

Methods

The proposed approach utilizes a generative model based on distance-based ordering. The core assumption is that a worker's response is a noisy observation of the true order, with the probability of a specific ranking decreasing as its distance from the truth increases.

The Probabilistic Model

The model incorporates a worker ability parameter, $\lambda^{(k)}$, for each worker $k$. A high $\lambda$ indicates an expert whose responses cluster tightly around the ground truth, while a low $\lambda$ indicates a worker whose responses are closer to random noise. By employing the Spearman distance (the Euclidean distance between rank vectors), the researchers treat rankings as coordinates in a high-dimensional vector space.

Overview of the item ordering task and the estimation framework.

Estimation Algorithm

To determine the true ordering and worker abilities, the authors use an iterative Maximum Likelihood Estimation (MLE) algorithm:

  1. Step A: Given current estimates of worker abilities, calculate the "weighted mean" of the ranks to estimate the true order.
  2. Step B: Given the estimated true order, optimize the worker ability parameters based on the distance of their individual responses from that truth.

The model was evaluated using two real-world datasets from the Lancers marketplace: word ordering (arranging words into sentences) and sentence ordering (restoring logical flow to a paragraph).

Results

The researchers compared their distance-based model against a baseline that aggregates rankings without weighting for worker ability.

Performance and Redundancy

A key finding is the "redundancy threshold." The sophisticated quality control model begins to significantly outperform the baseline once the number of workers per task exceeds 5 or 6. Below this threshold, the data is often too sparse for the "intelligence" of the model to provide a meaningful advantage.

Comparison of averaged squared error between the proposed model and baseline as the number of workers increases.

Accuracy Gains

As shown in the experimental results, the proposed method consistently achieves a lower Averaged Squared Error (Spearman distance from ground truth) compared to the baseline. The model effectively filters noise in both word-level and sentence-level tasks, demonstrating the versatility of the distance-based approach across different ordering problems. By treating ability as a continuous parameter rather than a binary classification, the model provides a more nuanced weighting of every contribution.

Discussion and Conclusion

While the model is effective for small sets (5-6 items), the factorial growth of the search space ($N!$) presents computational challenges for larger tasks. Future work may require heuristic search methods to handle dozens of items. Additionally, the current model assumes static worker ability; incorporating temporal changes or task-specific expertise could further improve accuracy. Finally, exploring alternative distance metrics like Kendall’s Tau might better capture specific human error patterns. Despite these limitations, the framework provides a robust solution for aggregating complex crowd-sourced rankings.

Summary updated

Abstract

One of the biggest challenges in crowdsourcing is quality control which is to expect high quality results from crowd workers who are not necessarily very capable nor motivated.In this paper, we consider item ordering questions, where workersare asked to arrange multiple items in the correct order. We propose a probabilistic generative model of crowd answers by extending a distance-based order model to incorporate worker ability, and give an efficient estimation algorithm.
BibTeX
@inproceedings{matsui2013crowdsourcing,
  author = {Toshiko Matsui and Yukino Baba and Toshihiro Kamishima and Hisashi Kashima},
  title = {{Crowdsourcing Quality Control for Item Ordering Tasks}},
  booktitle = {HCOMP (Works in Progress / Demos)},
  year = {2013},
}
Figure