Learning to Truncate Ranked Lists for Information Retrieval

Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Xueqi Cheng

Introduction

In Information Retrieval (IR), systems traditionally focus on ranking documents by relevance. However, in professional domains such as legal or patent search, providing a ranked list is often insufficient; users require a definitive "stopping point" to balance the cost of review against the risk of missing critical information. This task, known as Ranked List Truncation, aims to find the optimal cutoff that maximizes utility metrics like the F1-score.

Existing methods typically treat truncation as a sequential decision process, which can suffer from local bias and indirect optimization of the target metrics. This paper introduces AttnCut, a global, attention-based approach that directly optimizes for user-defined rewards.

Proposed Architecture

AttnCut moves away from sequential "stop-or-go" decisions toward a global perspective using a Transformer-based architecture. The model consists of three primary layers:

  1. Encoding Layer: A two-layer Bi-LSTM captures local statistical features for each document, including relevance scores and basic document metadata.
  2. Attention Layer: This is the core innovation. A single-layer Transformer (self-attention) allows every document in the list to "see" every other document. This provides a global view of the list's quality distribution, enabling the model to identify the optimal cutoff based on the entire context.
  3. Decision Layer: An MLP followed by a Softmax outputs a probability distribution over all possible cutoff positions.
The AttnCut architecture featuring Encoding, Attention, and Decision layers.

Optimization via RAML

To bridge the gap between training objectives and evaluation metrics, AttnCut utilizes Reward Augmented Maximum Likelihood (RAML). Unlike standard Maximum Likelihood Estimation (MLE), which only rewards the absolute ground-truth cutoff, RAML encourages the model to prefer positions with high rewards (e.g., high F1 scores) by defining a target distribution based on actual metric performance. This "smooths" the learning process and improves stability compared to Reinforcement Learning (RL) approaches.

Experimental Setup

The model was evaluated on two standard IR benchmarks: Robust04 (250 queries) and MQ2007 (1692 queries). Initial ranked lists were generated using both probabilistic (BM25) and deep learning (DRMM) retrieval models to test AttnCut's robustness across different ranking signals.

Experimental Results

AttnCut consistently outperforms previous state-of-the-art methods, including BiCut and the Transformer-based Choppy. On the Robust04 dataset using BM25, AttnCut achieved a 12.26% relative improvement in F1 score over BiCut.

Performance comparison of AttnCut against various baseline methods on Robust04 and MQ2007 datasets.

Ablation studies confirmed that the combination of the attention mechanism and RAML optimization is critical. The RAML-based version outperformed variants using standard MLE or RL, demonstrating superior alignment with the evaluation metrics.

Discussion and Insights

The success of AttnCut highlights two key findings:

  • Global Context is Essential: The shift to a Transformer architecture suggests that the optimal stopping signal depends on how a document's relevance compares to the overall density of relevant documents in the list, rather than just its individual score.
  • Metric-Direct Learning: RAML effectively bridges the training-evaluation gap by rewarding "near-optimal" decisions that still yield high utility for the end-user.

Qualitative analysis revealed a "conservative bias" in very long lists (rank 250+), where the model becomes hesitant to truncate as relevant document density becomes sparse. Future work may involve incorporating richer document features, such as topical diversity or embeddings, to better distinguish between redundant and unique information.

Conclusion

AttnCut represents a significant advancement in ranked list truncation by leveraging global attention and direct metric optimization. Its ability to handle recall constraints further enhances its practicality for high-stakes professional search scenarios where missing information is costly.

Summary updated

Abstract

Ranked list truncation is of critical importance in a variety of professional information retrieval applications such as patent search or legal search. The goal is to dynamically determine the number of returned documents according to some user-defined objectives, in order to reach a balance between the overall utility of the results and user efforts. Existing methods formulate this task as a sequential decision problem and take some pre-defined loss as a proxy objective, which suffers from the limitation of local decision and non-direct optimization. In this work, we propose a global decision based truncation model named AttnCut, which directly optimizes user-defined objectives for the ranked list truncation. Specifically, we take the successful transformer architecture to capture the global dependency within the ranked list for truncation decision, and employ the reward augmented maximum likelihood (RAML) for direct optimization. We consider two types of user-defined objectives which are of practical usage. One is the widely adopted metric such as F1 which acts as a balanced objective, and the other is the best F1 under some minimal recall constraint which represents a typical objective in professional search. Empirical results over the Robust04 and MQ2007 datasets demonstrate the effectiveness of our approach as compared with the state-of-the-art baselines.
BibTeX
@inproceedings{wu2021learning,
  author = {Chen Wu and Ruqing Zhang and Jiafeng Guo and Yixing Fan and Yanyan Lan and Xueqi Cheng},
  title = {{Learning to Truncate Ranked Lists for Information Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i5.16572},
}
Figure