Abstract
Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scalar metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.
Introduction
In Information Retrieval (IR), the standard practice for evaluating systems involves distilling a ranked list of results into a single scalar value. Metrics like Average Precision (AP), Normalized Discounted Cumulative Gain (NDCG), and Reciprocal Rank (RR) are ubiquitous. However, these metrics suffer from two primary limitations:
- Low Label Efficiency: Most metrics employ rank-based discounting (e.g., $1/\log(\text{rank})$) to prioritize top-tier results. While this mirrors common user behavior, it effectively discards information about relevant items deeper in the list. Improvements at lower ranks are often mathematically invisible because the contribution of, say, the 10th relevant item is negligible compared to the 1st.
- Rigid User Assumptions: Metrics bake in specific assumptions about what a "typical" user wants. For instance, RR assumes a user seeks only one relevant item. This fails to account for "high-recall" scenarios, such as legal discovery or systematic literature reviews, where finding every relevant document is critical.
Recall-Paired Preference (RPP)
To address these issues, we propose Recall-Paired Preference (RPP). Rather than assigning an absolute score to a single system, RPP performs a direct comparison between two ranked lists by simulating a diverse set of user "pseudo-populations" with varying recall needs.
How RPP Works
Instead of calculating a difference in scalar metrics ($Δ = μ(π) - μ(π')$), RPP evaluates the preference directly from the rankings.
- Pseudo-Populations: The method simulates users satisfied by different numbers of relevant items ($i=1, 2, \dots, R$).
- Search Length ($f_i$): For each recall level $i$, we measure the rank position of the $i$-th relevant item. If System A places the $i$-th relevant item at a higher rank (lower index) than System B, System A is preferred for that specific sub-population.
- Preference Aggregation: RPP aggregates these preferences across all possible recall levels. By giving every recall level equal weight, RPP ensures that improvements at any depth of the ranking are accounted for, significantly increasing sensitivity.
Experimental Validation
We validated RPP across a broad spectrum of tasks, including ad-hoc search (TREC Common Core, Deep Learning, Robust 2004), web search (TREC Web track), and recommendation (MovieLens 1M, LibraryThing, BeerAdvocate).
Key Findings
- High Correlation with Baselines: RPP maintains strong alignment with industry standards, correlating with AP ($r \approx 0.86$) and NDCG ($r \approx 0.87$). It generally agrees with established metrics while providing finer-grained resolution.
- Superior Discriminative Power: RPP consistently identifies significant differences between systems that traditional metrics miss. On the Robust 2004 dataset, RPP detected significant wins in 63% of system pairs, compared to only 53% for AP.
- Robustness to Sparsity: Despite using more of the available labels, RPP is remarkably stable. Even when 50-90% of relevance labels are removed, its performance remains as robust as NDCG, which is famously resilient to incomplete data.
Conclusion and Future Work
RPP offers a more sensitive and flexible alternative to traditional scalar metrics by bridging the gap between offline evaluation and online preference-based methods like interleaving. While RPP excels at discrimination, future work will explore how to directly optimize machine learning models for this preference-based objective. Additionally, we aim to refine the "pseudo-population" models by weighting recall levels based on real-world behavioral data, further aligning offline evaluation with actual user utility.
Summary updated