Introduction
Crowdsourced relevance judgments—human-labeled data indicating whether a webpage satisfies a search query—are fundamental to Information Retrieval (IR). Traditionally, platforms like Amazon Mechanical Turk distribute these tasks randomly, often pairing workers with technical or niche topics outside their expertise. This mismatch can lead to poor data quality. This paper investigates whether allowing workers to choose their tasks improves the alignment between worker knowledge and task requirements, leading to more reliable judgments.
Methodology
The researchers conducted a controlled experiment with 26 subjects to compare two task-routing conditions:
- Choice Condition: Subjects were presented with five search queries and selected one to judge.
- Yoked Condition: To control for query difficulty, each "yoked" subject was assigned the exact query previously selected by a "choice" subject.
Participants performed 15 tasks, each involving the relevance assessment of five webpages on a 5-point scale. To measure the stability of worker understanding, the study utilized pre- and post-judgment questionnaires where workers rated their expertise, interest, and confidence, and described the perceived search intent.
Data Selection
Queries were sampled from major search engine logs and balanced for length, topic, and click entropy (a measure of ambiguity). For each query, five image captures of search results were provided to the judges.
Analysis of Choice as an Expertise Filter
The study confirms that when given the option, workers naturally gravitate toward topics they understand. This self-selection acts as an implicit expertise filter.
As shown in the table above, popular or general-interest queries like "soccer" are frequently selected, while highly specific or technical queries like "th50px500u" are often bypassed. This "non-selection" signal is valuable; it identifies tasks that may require specialized knowledge or higher incentives to attract qualified judges.
Furthermore, the researchers used "intent stability" as a proxy for expertise. If a worker significantly alters their description of a searcher's intent after seeing the results, it suggests they lacked an initial understanding of the query. Choice subjects demonstrated much higher stability, modifying their intent assessments only 15.38% of the time, compared to 29.23% for the yoked group.
Key Results
The experimental data highlights significant advantages for the choice condition across several self-reported metrics.
- Expertise: Choice subjects rated their expertise 0.30 points higher (on a 4-point scale) before judging and 0.41 points higher after judging than the yoked group.
- Interest & Confidence: Choice subjects reported significantly higher interest (+0.33) and greater confidence in their initial assessments of query intent.
- Performance Stability: The lower rate of intent modification among choice subjects suggests they possessed a clearer preconceived notion of the task, leading to more consistent judgments.
Conclusion and Future Directions
While this lab-scale study demonstrates that choice improves worker confidence and task alignment, further research is needed to validate these findings in large-scale, noisy crowdsourcing environments. A key question remains: does increased confidence and stability directly translate to higher "ground truth" accuracy in relevance labels?
Future routing systems could leverage "selection ratios" to identify difficult tasks. Queries that are repeatedly ignored could be flagged for specialists or assigned higher rewards, ensuring that knowledge-intensive tasks are handled by the most capable workers.
Summary updated