Relevance dimensions in preference-based IR evaluation

Jin Young Kim, Gabriella Kazai, Imed ZitouniSIGIR 2013 · DOI

TL;DR

This paper explores how users judge the quality of search results when presented with two lists side-by-side. They found that user preferences go beyond simple topical relevance, incorporating dimensions like diversity, and that collecting these dimensional preferences improves judgment quality and better reflects user satisfaction than traditional metrics like NDCG.

What Problem Does This Paper Address?

Traditional information retrieval (IR) evaluation often relies on assessing individual documents for relevance to a query. While this method has its merits, it overlooks how users actually experience search results in the real world. Users don't just care if individual documents are relevant; they also consider the overall set of results. For instance, a list with diverse relevant results that cover multiple user intents might be preferred over a list with only redundant, albeit relevant, documents. This paper addresses the challenge of understanding and measuring these multi-dimensional aspects of relevance that influence user preferences when comparing entire result sets.

Methods

The authors propose and implement a method for collecting "dimensional preference labels." Instead of just asking users to pick which of two search result lists they prefer overall, they ask them to compare the lists along several specific dimensions: Relevance (topicality), Diversity, Authority, Freshness, and Caption Quality. They also collect overall preference judgments and traditional, isolated relevance judgments for individual documents. Experiments were conducted with professional assessors who compared search result lists from two different search engines. They used a 7-point scale for judgments and collected data for 523 queries.

Datasets

The study used 523 queries extracted from the query logs of a commercial search engine. For each query, the top 10 search results were scraped from two search engines (referred to as "Engine Baseline" and "Engine Experiment"). This resulted in a dataset of query-URL pairs and associated search results used for the preference judgment tasks. No new benchmark datasets were introduced; rather, existing search engine results were used.

Key Insights and Ideas

  • Beyond Topicality: User preferences for search result lists are not solely driven by how topically relevant individual documents are. Other factors play a significant role.
  • Diversity Matters: Diversity emerged as a crucial dimension, especially when topical relevance was similar between two result sets. This suggests that users value seeing a range of perspectives or answers.
  • Dimensional Judgments Improve Quality: Asking judges to consider specific dimensions of relevance leads to better-quality judgments, evidenced by higher inter-assessor agreement. This implies that breaking down the complex task of preference judgment into smaller, focused criteria helps assessors make more consistent and reliable decisions.
  • NDCG's Limitations: Traditional metrics like Normalized Discounted Cumulative Gain (NDCG), which are based on isolated document relevance, correlate well with topical relevance but fail to capture the nuances of overall user preference, particularly when diversity is a key factor. This highlights a potential gap between what IR systems are optimized for (topical relevance) and what truly satisfies users.

Key Results

  • The "Experiment" engine was generally preferred over the "Baseline" engine in overall preference judgments.
  • When topical relevance was tied between the two engines, Diversity had a significant impact on overall preference.
  • Inter-judge agreement (measured by Cohen's Kappa) was higher in the "treatment" group (which collected dimensional preferences) compared to the "control" group (which only collected overall preferences).
  • Spearman correlation analysis showed that Relevance was the most correlated dimension with overall preference, followed by Diversity.
  • Correlations between preference judgments and dNDCG (difference in NDCG scores) were higher for the treatment group, especially at lower ranks, suggesting preference labels are more holistic.
  • Relevance judgments showed the highest correlation with dNDCG, but other dimensions like Diversity and Caption Quality were poorly correlated, reinforcing the idea that NDCG doesn't capture these aspects.

Limitations and Future Work

  • Subjectivity of Dimensions: While the paper proposes specific dimensions, the interpretation and weighting of these dimensions can still be subjective for assessors.
  • Limited Scope of Dimensions: The authors explored five dimensions, but other aspects of relevance might also influence user preferences.
  • Future Work: The authors suggest further exploring the relationship between dimensional preferences and user satisfaction, as well as developing new evaluation metrics that can better capture these multi-dimensional aspects of relevance. They also hint at the potential for using dimensional preference data to guide system design and optimization beyond just topical relevance.

Summary updated

BibTeX
@inproceedings{kim2013relevance,
  author = {Jin Young Kim and Gabriella Kazai and Imed Zitouni},
  title = {{Relevance dimensions in preference-based IR evaluation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2013},
  doi = {10.1145/2484028.2484168},
}
Figure