The effects of choice in routing relevance judgments

Edith Law, Paul N. Bennett, Eric Horvitz

Introduction

Crowdsourced relevance judgments—human-labeled data indicating whether a webpage satisfies a search query—are fundamental to Information Retrieval (IR). Traditionally, platforms like Amazon Mechanical Turk distribute these tasks randomly, often pairing workers with technical or niche topics outside their expertise. This mismatch can lead to poor data quality. This paper investigates whether allowing workers to choose their tasks improves the alignment between worker knowledge and task requirements, leading to more reliable judgments.

Methodology

The researchers conducted a controlled experiment with 26 subjects to compare two task-routing conditions:

  1. Choice Condition: Subjects were presented with five search queries and selected one to judge.
  2. Yoked Condition: To control for query difficulty, each "yoked" subject was assigned the exact query previously selected by a "choice" subject.

Participants performed 15 tasks, each involving the relevance assessment of five webpages on a 5-point scale. To measure the stability of worker understanding, the study utilized pre- and post-judgment questionnaires where workers rated their expertise, interest, and confidence, and described the perceived search intent.

Data Selection

Queries were sampled from major search engine logs and balanced for length, topic, and click entropy (a measure of ambiguity). For each query, five image captures of search results were provided to the judges.

Analysis of Choice as an Expertise Filter

The study confirms that when given the option, workers naturally gravitate toward topics they understand. This self-selection acts as an implicit expertise filter.

Table 1: Examples of queries commonly selected vs. those frequently ignored by workers.

As shown in the table above, popular or general-interest queries like "soccer" are frequently selected, while highly specific or technical queries like "th50px500u" are often bypassed. This "non-selection" signal is valuable; it identifies tasks that may require specialized knowledge or higher incentives to attract qualified judges.

Furthermore, the researchers used "intent stability" as a proxy for expertise. If a worker significantly alters their description of a searcher's intent after seeing the results, it suggests they lacked an initial understanding of the query. Choice subjects demonstrated much higher stability, modifying their intent assessments only 15.38% of the time, compared to 29.23% for the yoked group.

Key Results

The experimental data highlights significant advantages for the choice condition across several self-reported metrics.

Figure 1: Self-ratings of expertise, interest, and confidence before and after judgments for Choice and Yoked conditions.
  • Expertise: Choice subjects rated their expertise 0.30 points higher (on a 4-point scale) before judging and 0.41 points higher after judging than the yoked group.
  • Interest & Confidence: Choice subjects reported significantly higher interest (+0.33) and greater confidence in their initial assessments of query intent.
  • Performance Stability: The lower rate of intent modification among choice subjects suggests they possessed a clearer preconceived notion of the task, leading to more consistent judgments.

Conclusion and Future Directions

While this lab-scale study demonstrates that choice improves worker confidence and task alignment, further research is needed to validate these findings in large-scale, noisy crowdsourcing environments. A key question remains: does increased confidence and stability directly translate to higher "ground truth" accuracy in relevance labels?

Future routing systems could leverage "selection ratios" to identify difficult tasks. Queries that are repeatedly ignored could be flagged for specialists or assigned higher rewards, ensuring that knowledge-intensive tasks are handled by the most capable workers.

Summary updated

Abstract

The emergence of human computation systems, including Mechanical Turk and games with a purpose, has made it feasible to distribute relevance judgment tasks to workers over the Web. Most human computation systems assign tasks to individuals randomly, and such assignments may match workers with tasks that they may be unqualified or unmotivated to perform. We compare two groups of workers, those given a choice of queries to judge versus those who are not, in terms of their self-rated competence and their actual performance. Results show that when given a choice of task, workers choose ones for which they have greater expertise, interests, confidence, and understanding.
BibTeX
@inproceedings{law2011effects,
  author = {Edith Law and Paul N. Bennett and Eric Horvitz},
  title = {{The effects of choice in routing relevance judgments}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2011},
  doi = {10.1145/2009916.2010082},
}
Figure