Learning to Truncate Ranked Lists for Information Retrieval

Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Xueqi Cheng

Abstract

Ranked list truncation is of critical importance in professional information retrieval applications such as patent or legal search. The goal is to dynamically determine the number of returned documents to balance result utility with user effort. Existing methods often rely on sequential decision-making and proxy objectives, leading to local optima. We propose AttnCut, a global decision-based truncation model that leverages the Transformer architecture to capture dependencies across the entire ranked list. By employing Reward Augmented Maximum Likelihood (RAML), AttnCut directly optimizes user-defined objectives like the F1-score. Empirical results on the Robust04 and MQ2007 datasets demonstrate that AttnCut significantly outperforms state-of-the-art baselines.

Introduction

In traditional Information Retrieval (IR), the primary focus is on ranking relevant documents at the top of a list. However, for professional users—such as patent attorneys or medical researchers—the length of the returned list is as crucial as the order. Every additional document requires manual review, incurring costs in time and cognitive effort.

Truncation aims to find the optimal "cut-off" point where the benefit of finding more relevant documents (recall) no longer outweighs the cost of reviewing non-relevant ones (precision). While prior methods like BiCut treat this as a sequential "stop-or-go" decision, they lack a global view of the list, potentially missing clusters of relevant documents further down the ranking.

The AttnCut Framework

AttnCut shifts the paradigm from local, sequential decisions to a global optimization approach. The model architecture is designed to process the entire top-$N$ ranked list simultaneously to determine the most effective truncation point.

1. Model Architecture

The framework consists of three primary layers:

  • Encoding Layer: Uses a Bi-directional LSTM (Bi-LSTM) to capture local features for each document, such as its initial relevance score and position-specific metadata.
  • Attention Layer: A Transformer-based self-attention mechanism allows every document in the list to interact with every other document. This global context ensures the decision at rank $k$ is informed by the distribution of relevance across the entire candidate set.
  • Decision Layer: A final Multi-Layer Perceptron (MLP) and Softmax layer output a probability distribution over all possible cut-off positions.
The AttnCut model architecture featuring encoding, attention, and decision layers.

2. Direct Optimization via RAML

Standard training typically uses Maximum Likelihood Estimation (MLE), which penalizes any prediction that isn't the exact ground-truth cut-off. AttnCut instead utilizes Reward Augmented Maximum Likelihood (RAML). RAML smooths the target distribution based on a reward function (e.g., the F1-score). This allows the model to learn that a cut-off near the optimal point is "better" than one far away, directly aligning the training objective with the evaluation metric users care about.

Experimental Setup

The model was evaluated using two benchmark datasets:

  • Robust04: A TREC dataset with 250 queries and ~528k documents, characterized by a high density of relevant documents per query (~70).
  • MQ2007: A LETOR benchmark with 1,692 queries, averaging roughly 10 relevant documents per query.

To test robustness, AttnCut was applied to lists generated by two different initial rankers: the traditional probabilistic BM25 model and the deep learning-based DRMM (Deep Relevance Matching Model).

Performance Evaluation

AttnCut consistently outperformed state-of-the-art baselines, including fixed-length truncation, greedy approaches, and sequential models like BiCut and Choppy.

Comparison of AttnCut against state-of-the-art baselines on Robust04 and MQ2007 datasets.

Key findings from the evaluation include:

  • Significant Gains: On the Robust04 dataset using BM25, AttnCut achieved a 12.26% relative improvement in F1-score over BiCut.
  • Global Context Advantage: Qualitative analysis showed that AttnCut's predicted cut-off positions closely mirrored the "Oracle" (theoretical perfect) cut-offs, particularly in lists where relevant documents were sparsely distributed.
  • Recall-Constraint Flexibility: A variant of AttnCut was developed for scenarios requiring a minimum recall threshold (e.g., "find 70% of all relevant documents"). By predicting recall bins, the model successfully balanced precision while meeting strict professional recall requirements.

Conclusion

AttnCut demonstrates that ranked list truncation is fundamentally a global optimization problem. By combining the contextual power of Transformers with the metric-direct optimization of RAML, the model provides a robust solution for professional search tasks. Future work will explore incorporating document diversity features and optimizing the model for high-scale, real-time retrieval environments.

Summary updated

Abstract

Ranked list truncation is of critical importance in a variety of professional information retrieval applications such as patent search or legal search. The goal is to dynamically determine the number of returned documents according to some user-defined objectives, in order to reach a balance between the overall utility of the results and user efforts. Existing methods formulate this task as a sequential decision problem and take some pre-defined loss as a proxy objective, which suffers from the limitation of local decision and non-direct optimization. In this work, we propose a global decision based truncation model named AttnCut, which directly optimizes user-defined objectives for the ranked list truncation. Specifically, we take the successful transformer architecture to capture the global dependency within the ranked list for truncation decision, and employ the reward augmented maximum likelihood (RAML) for direct optimization. We consider two types of user-defined objectives which are of practical usage. One is the widely adopted metric such as F1 which acts as a balanced objective, and the other is the best F1 under some minimal recall constraint which represents a typical objective in professional search. Empirical results over the Robust04 and MQ2007 datasets demonstrate the effectiveness of our approach as compared with the state-of-the-art baselines.
BibTeX
@inproceedings{wu2021learning,
  author = {Chen Wu and Ruqing Zhang and Jiafeng Guo and Yixing Fan and Yanyan Lan and Xueqi Cheng},
  title = {{Learning to Truncate Ranked Lists for Information Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i5.16572},
}
Figure