Abstract
Learning to rank for information retrieval has gained a lot of interest in the recent years but there is a lack for large real-world datasets to benchmark algorithms. That led us to publicly release two of the largest "Learning to Rank" (LTR) datasets to date and organize the 2010 Yahoo! Learning to Rank Challenge. This paper summarizes the challenge, revealing that while complex ranking-specific loss functions are popular in literature, ensemble decision tree methods—and even simple regression—reign supreme when applied to real-world data at scale.
Introduction: Addressing the Scale Gap
For years, the field of Learning to Rank (LTR) was hampered by a "scale gap." Researchers primarily used the LETOR benchmark, which contained only about 100 queries. In contrast, commercial search engines like Yahoo!, Google, and Microsoft process millions of queries. Because conclusions drawn from tiny datasets often fail to hold up at production scales, and because search companies rarely shared their proprietary data, academic progress was often disconnected from industrial reality.
The authors addressed this by publicly releasing two massive internal datasets and organizing a global competition to benchmark state-of-the-art algorithms on "real" data.
Methods
The challenge featured two tracks: a Standard Track for learning a ranking function for the US market (Set 1), and a Transfer Learning Track for a smaller market (Set 2) leveraging the US data. Participants utilized various LTR methods categorized by their treatment of the ranking problem:
- Pointwise: Treating ranking as a standard regression or classification problem for each individual document.
- Pairwise: Focusing on the relative order of pairs of documents (e.g., Document A is better than Document B).
- Listwise: Optimizing the entire ordered list of documents for a query simultaneously.
Datasets
The released datasets were subsets of Yahoo!'s production data. Set 1 (US) included nearly 20,000 training queries and 473,000 documents, while Set 2 (Asian Country) served as the transfer learning target. Each query-document pair featured approximately 700 anonymized features, including web graph metrics, document statistics, and text-match scores.
Human editors assigned relevance grades on a 5-point scale, from "Bad" (0) to "Perfect" (4). As shown in the label distribution, the datasets are highly imbalanced, reflecting the reality of web search where most documents are not highly relevant to a given query.
Key Insights and Ideas
- The "Regression Paradox": Surprisingly, simple squared-loss regression (GBDT) performed nearly as well as complex, ranking-specific loss functions. With a powerful enough model like an ensemble of trees, the specific choice of loss function becomes secondary to the model's capacity to fit the data.
- Ensembles are Essential: Every winning team used ensemble methods. LambdaMART (a boosted tree version of LambdaRank) emerged as a dominant force.
- The Transfer Learning Hurdle: Benefits of transfer learning were limited. Competitors struggled to significantly outperform models trained only on local data, suggesting search intent varies significantly across markets.
- Metric Nuance: While the two primary metrics—NDCG and ERR—were highly correlated (0.89), they disagreed on the very best submissions, highlighting the importance of metric selection for specific user behaviors.
Key Results
The challenge drew over 1,000 registered teams. The top team in Track 1 achieved an ERR of 0.46861. Differences between the top five finishers were incredibly small, often lacking statistical significance. Decision trees and ensemble techniques like boosting, bagging, and random forests were used by all winners.
Limitations and Future Work
Feature anonymization prevented participants from performing domain-specific feature engineering, a critical part of real-world machine learning. The lack of success in transfer learning suggests either dataset dissimilarity or that the "local" training set was already sufficiently large. Future research should move beyond query-document pairs to explore more complex user models reflecting the evolving nature of web search.
Summary updated