Abstract
Ranked list truncation is of critical importance in professional information retrieval applications such as patent or legal search. The goal is to dynamically determine the number of returned documents to balance result utility with user effort. Existing methods often rely on sequential decision-making and proxy objectives, leading to local optima. We propose AttnCut, a global decision-based truncation model that leverages the Transformer architecture to capture dependencies across the entire ranked list. By employing Reward Augmented Maximum Likelihood (RAML), AttnCut directly optimizes user-defined objectives like the F1-score. Empirical results on the Robust04 and MQ2007 datasets demonstrate that AttnCut significantly outperforms state-of-the-art baselines.
Introduction
In traditional Information Retrieval (IR), the primary focus is on ranking relevant documents at the top of a list. However, for professional users—such as patent attorneys or medical researchers—the length of the returned list is as crucial as the order. Every additional document requires manual review, incurring costs in time and cognitive effort.
Truncation aims to find the optimal "cut-off" point where the benefit of finding more relevant documents (recall) no longer outweighs the cost of reviewing non-relevant ones (precision). While prior methods like BiCut treat this as a sequential "stop-or-go" decision, they lack a global view of the list, potentially missing clusters of relevant documents further down the ranking.
The AttnCut Framework
AttnCut shifts the paradigm from local, sequential decisions to a global optimization approach. The model architecture is designed to process the entire top-$N$ ranked list simultaneously to determine the most effective truncation point.
1. Model Architecture
The framework consists of three primary layers:
- Encoding Layer: Uses a Bi-directional LSTM (Bi-LSTM) to capture local features for each document, such as its initial relevance score and position-specific metadata.
- Attention Layer: A Transformer-based self-attention mechanism allows every document in the list to interact with every other document. This global context ensures the decision at rank $k$ is informed by the distribution of relevance across the entire candidate set.
- Decision Layer: A final Multi-Layer Perceptron (MLP) and Softmax layer output a probability distribution over all possible cut-off positions.
2. Direct Optimization via RAML
Standard training typically uses Maximum Likelihood Estimation (MLE), which penalizes any prediction that isn't the exact ground-truth cut-off. AttnCut instead utilizes Reward Augmented Maximum Likelihood (RAML). RAML smooths the target distribution based on a reward function (e.g., the F1-score). This allows the model to learn that a cut-off near the optimal point is "better" than one far away, directly aligning the training objective with the evaluation metric users care about.
Experimental Setup
The model was evaluated using two benchmark datasets:
- Robust04: A TREC dataset with 250 queries and ~528k documents, characterized by a high density of relevant documents per query (~70).
- MQ2007: A LETOR benchmark with 1,692 queries, averaging roughly 10 relevant documents per query.
To test robustness, AttnCut was applied to lists generated by two different initial rankers: the traditional probabilistic BM25 model and the deep learning-based DRMM (Deep Relevance Matching Model).
Performance Evaluation
AttnCut consistently outperformed state-of-the-art baselines, including fixed-length truncation, greedy approaches, and sequential models like BiCut and Choppy.
Key findings from the evaluation include:
- Significant Gains: On the Robust04 dataset using BM25, AttnCut achieved a 12.26% relative improvement in F1-score over BiCut.
- Global Context Advantage: Qualitative analysis showed that AttnCut's predicted cut-off positions closely mirrored the "Oracle" (theoretical perfect) cut-offs, particularly in lists where relevant documents were sparsely distributed.
- Recall-Constraint Flexibility: A variant of AttnCut was developed for scenarios requiring a minimum recall threshold (e.g., "find 70% of all relevant documents"). By predicting recall bins, the model successfully balanced precision while meeting strict professional recall requirements.
Conclusion
AttnCut demonstrates that ranked list truncation is fundamentally a global optimization problem. By combining the contextual power of Transformers with the metric-direct optimization of RAML, the model provides a robust solution for professional search tasks. Future work will explore incorporating document diversity features and optimizing the model for high-scale, real-time retrieval environments.
Summary updated