Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing

Gabriella Kazai, Marijn Koolen, Jaap Kamps, Antoine Doucet, Monica LandoniINEX 2010 · DOI

TL;DR

This paper details the INEX 2010 Book Track, focusing on how crowdsourcing was used to overcome the significant bottleneck of creating large-scale test collections for evaluating book search systems. The authors demonstrate that with careful design, crowdsourcing can effectively generate both search topics and relevance judgments, paving the way for more scalable and cost-effective evaluation methodologies.

What Problem Does This Paper Address?

Evaluating information retrieval (IR) systems, especially for complex domains like digitized books, relies heavily on meticulously constructed test collections. These collections typically consist of search topics and corresponding relevance judgments (i.e., which documents are relevant to a given topic). However, creating these collections is a labor-intensive and expensive process. The INEX Book Track, aiming to advance techniques for searching, navigating, and reading digitized books, faced a critical challenge: building a sufficiently large test collection was a major bottleneck, requiring an unsustainable amount of effort from participants. This paper addresses this scalability problem by exploring and evaluating the use of crowdsourcing as a viable alternative to traditional methods for test collection construction.

Methods

The core methodological innovation presented in this paper is the application of crowdsourcing to two key aspects of test collection creation:

  • Topic Creation: Instead of relying solely on INEX participants, the researchers leveraged Amazon Mechanical Turk (AMT) to generate search topics. Two types of "Human Intelligence Tasks" (HITs) were designed:
  • Fact Finding Both in Books and in Wikipedia (Wiki HIT): Workers were tasked with finding factual statements that exist in both Wikipedia and the INEX book corpus.
  • Fact Finding in Books (Book HIT): Workers were asked to find factual statements within the INEX book corpus. The design of these HITs included clear instructions, example facts, and specific data to be collected (e.g., Wikipedia URL, book URL, page number). Payment was adjusted based on initial task completion rates and worker approval ratings.

  • Relevance Labeling: Crowdsourcing was also employed to gather relevance judgments for the created topics.

  • INEX Participants: Used a dedicated assessment system to search, browse, and annotate books, providing book and page-level relevance labels.
  • AMT Workers: Were presented with HITs containing pages from top-ranked search results. Each HIT included at least one page pre-labeled by an INEX participant to ensure exposure to relevant content. Three judgments per page were collected from AMT workers.

The paper also details strategies for pooling relevance labels from both sources to create a comprehensive and reliable "gold set" for evaluation. This involved combining different pooling strategies (rank-boosted and answer-boosted) and merging INEX and AMT labels, prioritizing INEX judgments in case of discrepancies.

Datasets

The primary dataset used is the INEX 2010 Book Track corpus, consisting of approximately 50,239 out-of-copyright digitized books. This corpus is in an XML format (BookML) derived from OCR text, with additional structural information.

The paper also implicitly uses Wikipedia as a source for topic creation in the "Wiki HIT."

A significant contribution is the newly constructed test collection for the Best Books (BB) and Prove It (PI) tasks, which includes:

  • Topics: A combination of topics created by INEX participants and crowdsourced topics from AMT.
  • Relevance Labels: A merged set of relevance judgments from both INEX participants and AMT workers, forming the official "ip2c-set" for evaluation.

Key Insights and Ideas

  • Crowdsourcing as a Scalability Solution: The central idea is that crowdsourcing, when properly designed, can effectively address the scalability challenges of creating large-scale IR test collections, significantly reducing the burden on domain experts.
  • Quality Control is Crucial: The paper emphasizes the importance of careful task design, worker selection (e.g., using approval rates), and payment strategies to ensure the quality of crowdsourced data.
  • Crowdsourcing for Both Topics and Judgments: The research demonstrates the feasibility of using crowdsourcing for both generating search topics and collecting relevance labels, offering a comprehensive approach to test collection construction.
  • AMT Quality Varies but Can Be Managed: While crowdsourced topics required significant filtering and modification, the quality of acceptable topics was often high and diverse. Similarly, relevance judgments showed good agreement with expert labels, especially with appropriate pooling and quality control.
  • Crowdsourcing Can Complement Expert Judgments: The methodology successfully integrated crowdsourced labels with expert judgments, creating a richer and more robust evaluation resource.

Key Results

  • Topic Creation: AMT workers contributed 58 topics, significantly more than the 25 from INEX participants. After filtering, 40 of the crowdsourced topics were deemed high quality, even more creative than those from INEX participants.
  • Relevance Labeling: A total of 12,275 relevance labels were collected (5,975 from INEX, 6,300 from AMT). Agreement between AMT majority vote and INEX gold set labels was high (0.72 for all classes).
  • Best Books (BB) Task: The best BB run achieved an NDCG@10 score of 0.6579, submitted by the University of California, Berkeley.
  • Prove It (PI) Task: The best PI run achieved an NDCG@10 score of 0.2946, submitted by the University of Amsterdam. This task, focusing on page-level retrieval for factual statements, showed surprisingly effective results.
  • Agreement Metrics: Table 3 shows high agreement and consensus among AMT workers (e.g., 0.90 consensus for all classes). Table 4 indicates that AMT workers were more likely to label pages as "refute" or "confirm" compared to INEX participants.

Limitations and Future Work

  • Crowdsourced Topic Quality Variability: The paper acknowledges that the quality of crowdsourced topics varies greatly and requires significant effort to filter out unsuitable submissions. Future work could focus on improving this through pre-selection of workers or more defensive task design.
  • Filtering Effort: While crowdsourcing scaled up topic creation, the filtering and modification of crowdsourced topics still required considerable effort.
  • Future Directions: The INEX Book Track plans to continue exploring crowdsourcing for test collection construction, aiming to remove the burden of relevance assessments from participants entirely in future iterations. The track will also shift focus to more social and semantic search scenarios, incorporating user-generated metadata and new tasks like "Social Search for Best Books" (SSBB).

Summary updated

BibTeX
@inproceedings{kazai2010overview,
  author = {Gabriella Kazai and Marijn Koolen and Jaap Kamps and Antoine Doucet and Monica Landoni},
  title = {{Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing}},
  booktitle = {INEX},
  year = {2010},
  doi = {10.1007/978-3-642-23577-1\_9},
}
Figure