A Comprehensive Dataset for Modern Learning to Rank Solutions (Abstract)

Domenico Dato, Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola TonellottoIIR 2022 · OpenReview

Introduction

In the field of Information Retrieval (IR), a significant divide has emerged between two primary methodologies for document ranking: traditional Learning-to-Rank (LtR) and modern Neural Learning-to-Rank. While both aim to improve search relevance, they have historically operated in separate silos due to data limitations. The introduction of the Istella22 dataset aims to bridge this gap, providing a unified platform for evaluating and combining these disparate approaches.

The "Great Divide" in Ranking Research

The evolution of search ranking has followed two distinct paths:

  1. Traditional Learning-to-Rank (LtR): These models, such as Gradient Boosted Decision Trees (GBDTs) like XGBoost or LightGBM, rely on "hand-crafted" feature vectors. These vectors represent specific signals like BM25 scores, PageRank, document length, and click-through rates.
  2. Neural Learning-to-Rank: Modern approaches leverage transformer-based architectures (e.g., BERT, Cross-Encoders) to process raw query and document text directly, capturing deep semantic relationships that hand-crafted features might miss.

The primary obstacle to comparative research has been the lack of a common dataset. Popular neural benchmarks like MS-MARCO provide raw text but lack the industrial-grade feature vectors required for traditional LtR. Conversely, classic LtR datasets often withhold raw text due to privacy or proprietary concerns, making it impossible to apply neural models.

Introducing Istella22

Istella22 is designed as the "missing link" for the IR research community. It provides a comprehensive corpus that supports both feature-based and text-based ranking models simultaneously. The dataset consists of:

  • 8.4 Million Web Documents: A large-scale corpus of crawled web content.
  • 2,198 Textual Queries: A diverse set of queries for evaluation.
  • 220 Hand-crafted Features: Industrial-grade signals derived from a real-world search engine (Istella).
  • 5-Graded Relevance Scale: Human-labeled judgments ranging from "not relevant" to "highly relevant."

By offering both raw text and pre-computed features, Istella22 enables a "fair fight" evaluation, allowing researchers to train different model types on identical data splits.

Key Insights: The Neural Lag and Synergy

Preliminary experiments conducted on Istella22 yielded surprising results that challenge the current dominance of neural-only approaches in academic literature.

The Performance Gap

In direct comparisons, traditional LtR models exploiting feature-based representations actually outperformed pre-trained transformer-based rankers. This "neural lag" suggests that raw semantic understanding, while powerful, may not yet fully capture the complex signals (such as link analysis or specific metadata) that industrial feature-based models utilize.

Hybrid Synergy

While neural models did not win outright, they proved to be highly effective as "feature generators." The most successful ranking strategy involved a hybrid approach: using a neural ranker to generate a semantic score and then feeding that score as an additional feature into a traditional LtR framework. This combination leverages the semantic depth of transformers alongside the robust signal processing of GBDTs.

Conclusion and Future Directions

Istella22 represents a significant step forward for the ranking research community, providing a realistic industrial context for model evaluation. The dataset highlights that the path to optimal search relevance likely lies in the synergy between traditional feature engineering and modern neural architectures.

Future research using Istella22 will likely explore more sophisticated Large Language Model (LLM) architectures to see if they can close the performance gap, as well as investigate the trade-offs between the high computational cost of neural models and the efficiency of feature-based systems. As one of the most comprehensive public resources available, Istella22 is poised to become a standard benchmark for the next generation of Learning-to-Rank solutions.

Summary updated

BibTeX
@inproceedings{dato2022comprehensive,
  author = {Domenico Dato and Sean MacAvaney and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
  title = {{A Comprehensive Dataset for Modern Learning to Rank Solutions (Abstract)}},
  booktitle = {IIR},
  year = {2022},
  url = {https://openreview.net/forum?id=7ExBjS_QXDa_},
}
Figure