Introduction
In Information Retrieval (IR), systems traditionally focus on ranking documents by relevance. However, in professional domains such as legal or patent search, providing a ranked list is often insufficient; users require a definitive "stopping point" to balance the cost of review against the risk of missing critical information. This task, known as Ranked List Truncation, aims to find the optimal cutoff that maximizes utility metrics like the F1-score.
Existing methods typically treat truncation as a sequential decision process, which can suffer from local bias and indirect optimization of the target metrics. This paper introduces AttnCut, a global, attention-based approach that directly optimizes for user-defined rewards.
Proposed Architecture
AttnCut moves away from sequential "stop-or-go" decisions toward a global perspective using a Transformer-based architecture. The model consists of three primary layers:
- Encoding Layer: A two-layer Bi-LSTM captures local statistical features for each document, including relevance scores and basic document metadata.
- Attention Layer: This is the core innovation. A single-layer Transformer (self-attention) allows every document in the list to "see" every other document. This provides a global view of the list's quality distribution, enabling the model to identify the optimal cutoff based on the entire context.
- Decision Layer: An MLP followed by a Softmax outputs a probability distribution over all possible cutoff positions.
Optimization via RAML
To bridge the gap between training objectives and evaluation metrics, AttnCut utilizes Reward Augmented Maximum Likelihood (RAML). Unlike standard Maximum Likelihood Estimation (MLE), which only rewards the absolute ground-truth cutoff, RAML encourages the model to prefer positions with high rewards (e.g., high F1 scores) by defining a target distribution based on actual metric performance. This "smooths" the learning process and improves stability compared to Reinforcement Learning (RL) approaches.
Experimental Setup
The model was evaluated on two standard IR benchmarks: Robust04 (250 queries) and MQ2007 (1692 queries). Initial ranked lists were generated using both probabilistic (BM25) and deep learning (DRMM) retrieval models to test AttnCut's robustness across different ranking signals.
Experimental Results
AttnCut consistently outperforms previous state-of-the-art methods, including BiCut and the Transformer-based Choppy. On the Robust04 dataset using BM25, AttnCut achieved a 12.26% relative improvement in F1 score over BiCut.
Ablation studies confirmed that the combination of the attention mechanism and RAML optimization is critical. The RAML-based version outperformed variants using standard MLE or RL, demonstrating superior alignment with the evaluation metrics.
Discussion and Insights
The success of AttnCut highlights two key findings:
- Global Context is Essential: The shift to a Transformer architecture suggests that the optimal stopping signal depends on how a document's relevance compares to the overall density of relevant documents in the list, rather than just its individual score.
- Metric-Direct Learning: RAML effectively bridges the training-evaluation gap by rewarding "near-optimal" decisions that still yield high utility for the end-user.
Qualitative analysis revealed a "conservative bias" in very long lists (rank 250+), where the model becomes hesitant to truncate as relevant document density becomes sparse. Future work may involve incorporating richer document features, such as topical diversity or embeddings, to better distinguish between redundant and unique information.
Conclusion
AttnCut represents a significant advancement in ranked list truncation by leveraging global attention and direct metric optimization. Its ability to handle recall constraints further enhances its practicality for high-stakes professional search scenarios where missing information is costly.
Summary updated