TL;DR
This paper introduces a novel "social game" approach, "Book Explorers' Competition," to gather relevance assessments for large digital book collections. By gamifying the process with incentives and collaborative elements, they aim to increase assessor engagement and improve the quality and exhaustiveness of relevance judgments, addressing the challenges of manual assessment for lengthy digital content.
What Problem Does This Paper Address?
Evaluating information retrieval (IR) systems traditionally relies on test collections, which include relevance assessments from human judges. However, as digital content like books and videos grows, traditional methods struggle. Assessing relevance for these items is time-consuming and effort-intensive due to their length, requiring extensive reading, browsing, and searching. This strain is amplified when needing passage-level relevance judgments for focused retrieval. Existing methods like pooling documents for judgment help, but the sheer scale of digital libraries still poses a significant challenge. The paper aims to tackle this by making the relevance assessment process more engaging and efficient through a collective, game-based approach.
Methods
The core of the paper's contribution is the Collective Relevance Assessment (CRA) method, implemented through a social game called the Book Explorers' Competition. This method involves three phases:
- Data Preparation and CRA Objectives: Candidate documents (books) and their relevant passages are identified using pooled search results from multiple IR runs. The goal is to collect assessments that identify coherent relevant passages and ensure consistent judgment of dispersed content.
- Game Design and Assessment System: The system is built around a client-server architecture with a user interface for content browsing, searching, and gameplay. The game itself is modeled as a two-player system with competing roles:
- Explorers: Locate and mark relevant content (passages or pages) within books.
-
Reviewers: Assess the quality of explorers' work by providing their own relevance judgments. Incentives are crucial: points are awarded for judgments, with bonus points for identifying relevant content and for "undisputed" relevant pages. This encourages both exploration and thorough review. The system also incorporates features for communication and dispute resolution between assessors.
-
Conducting the Game: The pilot study involved participants playing the game over several weeks.
A key innovation is the hybrid incentive structure that combines material rewards (for winners) with immaterial incentives like social status (through leaderboards) and community goals. The game design actively promotes communication and allows for self-re-assessment, aiming to eliminate trivial errors and encourage truthful assessments.
Datasets
The study utilized data from the INEX 2008 Book Track. This included:
- A corpus of over 50,000 digitized out-of-copyright books (approximately 17 million scanned pages).
- 70 TREC-style topics (title, description, narrative).
- Search results from 18 runs for the Book Retrieval (BR) task and 13 runs for the Page in Context (PiC) task.
The paper notes that the collected data from the pilot study itself forms a new benchmark for relevance assessments of digitized books.
Key Insights and Ideas
- Gamification for Engagement: A social game model can significantly boost assessor engagement and endurance for tedious tasks like relevance assessment, especially when incentives are well-aligned with desired behaviors.
- Community Incentive Matters: Participants with a direct stake in the outcome (e.g., INEX participants) were more motivated than those with only general interest, highlighting the importance of community incentives.
- Exploration vs. Reviewer Roles: The study found that participants were more inclined to explore than review, possibly due to perceived effort or the availability of content to review. This suggests a need to rebalance incentives or adjust the game flow.
- Assessor Behavior Varies: Judges exhibited different strategies (focused vs. exhaustive), impacting their reward accumulation and ranking. The "mixed" strategy, though less common, offered a good balance of engagement and reward.
- Relevance is Dispersed: Relevant content within books is often not concentrated but dispersed, requiring assessors to browse and search extensively, underscoring the need for robust browsing and search support in assessment systems.
- Quality Control through Self-Re-assessment: The game's design, which allows for re-assessment and dispute resolution, helps improve the quality of judgments and encourages assessors to reconsider their decisions based on feedback.
- Richness of Contextual Data: Annotations and interaction logs provide valuable insights into assessor behavior and can enrich the interpretation of relevance judgments.
Key Results
- Feasibility Demonstrated: The pilot study successfully engaged 17 participants, who collectively judged 3,478 books and 23,098 pages over an average of 11.4 hours per assessor.
- High Assessor Agreement: Pairwise agreement between judges was relatively high (around 80.7% for books, 57% for pages), exceeding levels reported in some previous TREC and INEX studies, likely due to the structured review and dispute resolution process.
- Exhaustive Assessment Achieved: The game design promoted assessor endurance, leading to a more exhaustive assessment of relevant content. Relevant books were explored more thoroughly (almost 9% of content) compared to non-relevant ones (less than 2%).
- Incentive Impact: Focused assessors earned more per book/minute but ranked lower, while mixed-strategy assessors achieved better rewards and rankings. Exhaustive judges, though most taxing, sometimes achieved the highest ranks.
- Data Collected: The system recorded a wealth of data, including highlighted regions, binary relevance labels, comments, navigational events, and relevance judgment events.
Limitations and Future Work
- Limited Engagement of Non-Community Members: Participants not directly benefiting from the collected data showed short-lived engagement, suggesting that community incentives are crucial for sustained participation.
- Role Imbalance: The lower uptake of the reviewer role and the delay in review availability present challenges. Future work could involve adjusting incentives for reviewers or separating exploration and review phases.
- Focus on Endurance vs. Focused Assessment: While the game promoted endurance, it didn't necessarily encourage the most focused assessment of relevant content. Future work should refine incentive structures to address this.
- Potential for Fraudulent Behavior: As with any incentive-based system, there's a risk of fraudulent behavior, which needs to be suppressed through careful design and quality control.
- Future Work: The authors plan to evaluate the quality of the collected relevance data against other assessment modes and test the reliability of the constructed test collections. They also suggest opening the system to allow users to contribute their own topics and exploring real-time dialogue between assessors.
Summary updated