TL;DR
This paper introduces the idea of requiring human annotators to provide short, text-based rationales for their relevance judgments in information retrieval tasks. They demonstrate that this simple addition significantly improves judgment quality and transparency without increasing task completion time for experienced workers, offering a win-win for both annotators and researchers.
What Problem Does This Paper Address?
Collecting high-quality human judgments for tasks like relevance assessment is notoriously difficult. Subjectivity, lack of insight into annotator reasoning, and the potential for "lazy" or even malicious annotators can lead to unreliable data. Traditional crowdsourcing platforms offer limited communication channels, making it hard to understand why an annotator made a particular decision, especially when that decision deviates from an expected "gold standard." This opacity hinders efforts to improve data quality and build trust in crowdsourced datasets.
Methods
The core innovation is the requirement for annotators to provide a "rationale" – in this case, a 2-3 sentence excerpt from a document that supports their relevance judgment. They experimented with three task designs:
- Standard Task: Collects only relevance judgments.
- Rationale Task: Collects relevance judgments and rationales, but the rationales themselves are not used for aggregation. This design isolates the effect of requiring a rationale.
- Two-Stage Task: A sequential process where one annotator provides a judgment and rationale, and then 1-4 subsequent reviewers confirm or modify it.
They also propose a novel method for filtering judgments based on the overlap of rationales provided by different annotators for the same item. If multiple annotators select very similar rationales, it suggests a higher degree of agreement and potentially better quality judgments.
Datasets
The study uses relevance judgments for the ClueWeb09 dataset, specifically drawing search topics and judgments from the 2009 TREC Web Track. They selected 700 documents and collected 5 crowd responses per webpage for each of their three task designs.
Key Insights and Ideas
- Rationales as a Transparency Tool: Requiring rationales makes the annotator's reasoning explicit, allowing for easier verification and understanding of subjective judgments. This is invaluable when annotators disagree with established "gold standards."
- Rationales Encourage Thoughtful Decisions: The act of formulating a rationale, even if it's just copying text, forces annotators to engage more deeply with the task, reducing the temptation to simply click through.
- No Significant Time Penalty for Experienced Workers: For annotators who have completed 20+ tasks, the time taken for the Rationale Task converges with the Standard Task. This suggests that experienced workers may already be performing a similar mental process implicitly.
- Rationale Overlap as a Quality Signal: The paper demonstrates a correlation between the similarity of rationales and judging accuracy, leading to a filtering mechanism that can further improve consensus quality.
- Two-Stage Task Amplifies Quality: Combining an initial judgment/rationale with subsequent review significantly boosts accuracy, even with fewer initial judgments compared to the Standard Task.
Key Results
- Improved Accuracy: The Rationale Task, even without using the rationales for aggregation, showed improved accuracy over the Standard Task. The Two-Stage Task yielded even higher quality judgments. For example, with 5 judges, the Rationale Task achieved 92% binary accuracy and 80% ternary accuracy, compared to 86% and 75% for the Standard Task.
- Filtering Benefits: Using their rationale overlap filtering method (THRESHOLD FILTERING) on aggregated judgments increased binary accuracy from 92% to 96% and ternary accuracy from 84% to 91%.
- Cost-Benefit: Experienced workers showed no significant increase in task completion time when providing rationales.
- Two-Stage Efficiency: The Two-Stage Task with just 2 judgments (initial + 1 reviewer) matched the performance of the Standard Task with 5 judgments, while also achieving higher ternary agreement. Reviewers corrected initial errors 82% of the time.
Limitations and Future Work
- Limitations: The study focused on experienced workers for the time analysis. The impact on novice workers might differ. The rationale filtering method is heuristic and could be further refined.
- Future Work: The authors plan to explore multi-stage iteration beyond two stages, dynamic collection of judgments based on rationale overlap, and using crowdsourced labels for A/B testing. They also aim to investigate dual-supervision for aggregation using rationales.
Summary updated