Offline Retrieval Evaluation Without Evaluation Metrics

Fernando Diaz, Fernando Diaz 0001, Andres Ferraro

TL;DR

Traditional search and recommendation metrics (like NDCG or Average Precision) often ignore the needs of users seeking high recall by over-discounting lower-ranked items. This paper introduces Recall-Paired Preference (RPP), a metric-free approach that compares systems by simulating diverse user groups with different recall needs, resulting in a much more sensitive and robust way to tell which system is actually better.

What Problem Does This Paper Address?

In the world of Information Retrieval (IR), we usually evaluate systems by distilling a ranked list of results into a single number (a scalar metric). Common metrics like Average Precision (AP) or Normalized Discounted Cumulative Gain (NDCG) are the industry standards.

However, the authors argue these metrics have two major flaws:

  1. Low Label Efficiency: Most metrics use "discounting" (like $1/\log(\text{rank})$) to prioritize the top results. While this reflects how users often behave, it effectively throws away information about relevant items further down the list. If two systems perform identically at rank 1 but one is much better at rank 20, many metrics will treat them as nearly equal.
  2. Rigid User Assumptions: Metrics bake in assumptions about what a "typical" user wants. For example, Reciprocal Rank (RR) assumes a user only wants one relevant item. This fails to account for "high-recall" scenarios, like a lawyer performing discovery or a researcher doing a systematic review, where finding every relevant document matters.

Methods

The authors propose Recall-Paired Preference (RPP). Instead of giving System A a score of 0.7 and System B a score of 0.6, RPP looks at the two ranked lists and asks: "For a user who wants to find exactly $i$ relevant items, which system requires less effort?"

Key Steps:

  • Pseudo-Populations: The method simulates different groups of users. One group might be satisfied with 1 relevant item, another wants 5, and another wants all of them.
  • Search Length ($f_i$): This is the rank position of the $i$-th relevant item. If System A puts the 3rd relevant item at rank 10 and System B puts it at rank 15, System A is preferred for that specific "recall level."
  • Preference Aggregation: RPP calculates the "sign" of the difference across all possible recall levels. If System A is better for more recall levels than System B, it wins the preference.
  • Metric-Free: It bypasses the need to calculate a standalone "quality score" for a single system, focusing instead on the direct competition between two systems.

Datasets

The authors validated RPP across a massive variety of tasks:

  • Search (TREC): Data from the Common Core tracks, Deep Learning tracks (Document and Passage ranking), and the classic Robust 2004 dataset.
  • Web Search: Six years of TREC Web track data, which includes subtopic judgments (evaluating how well a system covers different aspects of a query).
  • Recommendation: Three popular datasets—MovieLens 1M, LibraryThing, and BeerAdvocate—to ensure the method works for "suggested items" as well as "searched items."

Key Insights and Ideas

  • The "Aha" Moment on Discounting: The authors show that in traditional metrics, the contribution of the 10th relevant item is often mathematically negligible compared to the 1st. RPP fixes this by giving every "recall level" equal weight, ensuring that improvements deep in the ranking are actually counted.
  • Sensitivity is Key: Because RPP uses more of the available relevance labels (it doesn't "mute" the bottom of the list), it has much higher discriminative power. It can tell two systems apart even when traditional metrics say the difference is statistically insignificant.
  • Bridging Online and Offline: RPP is conceptually similar to "interleaving" (a common online A/B testing technique), but it is designed to work offline with static relevance judgments.

Key Results

  • High Correlation: RPP isn't just "different"—it's "better and consistent." It correlates strongly with AP ($r \approx 0.86$) and NDCG ($r \approx 0.87$), meaning it generally agrees with the "gold standards" but provides more detail.
  • Superior Discriminative Power: In almost every dataset, RPP detected significantly more "wins" between systems than AP, NDCG, or RR. For example, on the Robust dataset, RPP identified significant differences in 63% of system pairs, while AP only found them in 53%.
  • Robustness: Even when the authors intentionally "broke" the data (removing 50-90% of the relevance labels), RPP remained as stable as NDCG, which is famously robust to incomplete data.

Limitations and Future Work

  • Optimization: Most machine learning models are trained to optimize for specific metrics like NDCG. The authors note that it isn't yet clear how to directly "train" a model to maximize RPP, as it is a preference-based comparison rather than a single-system score.
  • Refining User Models: The current version of RPP weights all recall levels equally. Future work could involve using real-world behavioral data to weight these "pseudo-populations" more realistically (e.g., weighting a "1-item" user more than a "50-item" user for web search).
  • Computational Complexity: While not discussed as a major hurdle, aggregating preferences across all pairs of systems in a large-scale leaderboard is slightly more complex than simply sorting by a single scalar value.

Summary updated

Abstract

Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scalar metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.
BibTeX
@inproceedings{diaz2022offline,
  author = {Fernando Diaz and Fernando Diaz and Andres Ferraro},
  title = {{Offline Retrieval Evaluation Without Evaluation Metrics}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
  doi = {10.1145/3477495.3532033},
}

Loading related content...

Figure